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Executive summary 


The project 


This Learner Response System (LRS) intervention involves the use of electronic handheld devices that 
allow teachers and pupils to provide immediate feedback during lessons. For example, pupils can 
respond to a question using the device and responses are immediately visible to the teacher, or they 
can work through problems on the device at their own pace with answers provided as they go. The aim 
is to improve outcomes by increasing the speed and quality of classroom feedback. 


A team from Edge Hill University developed the intervention and trained teachers to deliver it to pupils 
in Years 5 and 6 in primary schools with higher than average proportions of children ever eligible for 
free school meals. The devices were to be used in at least three lessons a week for between 25 and 
32 weeks each year. The intervention ran from June 2014 to June 2016. One cohort of pupils used the 
LRS for two school years (‘cohort B’), and one cohort for only one school year (‘cohort A’). 


A cluster randomised controlled trial was used to evaluate the impact of the intervention on Year 6 
maths and reading outcomes. Around 6,500 pupils in 97 schools took part. The accompanying process 
evaluation involved classroom observations, teacher interviews, and focus groups with pupils. The 
project was funded as part of the EEF Digital Technology funding round in collaboration with the 
Nominet Trust. 


Key conclusions 


1. The project found no evidence that the LRS improved Key Stage 2 results in maths and 
reading for either cohort. 


2. The project found no evidence that the LRS improved the average scores of boys, girls or 
pupils who have ever been eligible for Free School Meals. 


3. Classroom teachers and children were generally positive about the LRS. Teachers 
welcomed the ability to quickly assess pupil responses for certain types of tasks and give 
instant feedback. They felt that the LRS helped to engage pupils and allowed different pupils 
to work at their own pace. 


4. Some teachers and pupils felt that the inability of the system to let children ‘show their 
working’ was a weakness. There was also a concern that incorrectly programmed or over- 
specified answers meant that pupils occasionally received negative feedback when in fact 
their answer was wholly or partially correct. There was some criticism from pupils and 
teachers that the small size of the handset made it difficult to type properly. 


5. The intervention was variably implemented, with a number of teachers not meeting the 
weekly targets for usage. Reasons given for this included staffing issues, lack of time, and 
the inability to use the system to its full potential. However, even in schools with high usage, 
the analysis did not find evidence of an impact. 


EEF security rating 


The security of the result for the two-year cohort is very high. The security of the result for the one-year 
cohort is moderate. The study was a well-designed cluster randomised controlled trial. The sample size 
was large, involving 97 schools and 6,572 pupils across two cohorts, and the trial was well-powered. 
For the one-year cohort, 11% and 12% of pupils who started the trial were not included in the final 
analysis (for maths and reading respectively), and there were some important differences between 
pupils in LRS schools and those in other schools in terms of prior attainment resulting in a security 
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rating of three padlocks. For the two-year cohort, only 7% and 10% of starting pupils were not included 
in the final analysis (for maths and reading respectively) and LRS pupils were similar to those in the 
comparison schools resulting in a security rating of five padlocks. 


Additional findings 


The main finding is that the LRS intervention did little to improve pupils’ Key Stage 2 test scores, 
regardless of whether the intervention was delivered over one or two years. There was no evidence of 
any positive impact for any pre-specified sub-group (boys, girls, or pupils ever eligible for FSM ). 
Schools that delivered medium doses of the LRS intervention did not perform better on the primary 
outcome measures than the control group, and even for high dose schools there is insufficient evidence 
to conclude that the intervention had a positive effect. 


These results run counter to those from a previous study involving similar electronic handsets used for 
a more limited time (12 weeks) in one subject. There are a variety of possible reasons for this. It might 
be because it was difficult to sustain the effective use of the devices over a broader range of topics for 
a longer time, or because in this study outcomes were not measured immediately after the intervention 
and were tested using high stakes national tests in reading and maths rather than focused tests 
designed to measure a specific set of skills for which the handsets were being used. 


Cost 


The full LRS package for one class costs around £7000, or £76.70 per pupil per year when averaged 
over 3 years. Running costs after the first year are low. Schools also need to meet the cost of staff 
cover for 7 days of teacher training in the first year and 2 in the second. 


Table 1: Summary of impact on primary outcome 


(eT colU) oN O]Ui vexe)nnl =) Effect size p-value Estimated i Coe) EEF EEF 
(heyemerey nlite (=alex=) months’ pupils Cost security 
a ale1)) progress eclalare) eelalaye| 
0.00 
Maths (cohort A) 0.96 0 months 2,837 |£LELEE 44a 
(-0.13 to 0.14) 
0.00 
Reading (cohort A) 0.96 0 months 2,829 LEEELE 44a 
(-0.12 to 0.13) 
-0.08 
Maths (cohort B) 0.24 -1 months 3,127 |££LEEE 4aaa8 
(-0.21 to 0.05) 
-0.04 
Reading (cohort B) 0.60 0 months 3,013 L£EEEE 44008 
(-0.17 to 0.10) 
-0.01 /a 
Maths teenies 0.94  Omonths 1677 E£f58 " 
ever eligible for FSM) (-0.15 to 0.14) 

i -0.01 /a 
meson \epnel tye 0.85  Omonths 1671 E£ee8 : 
ever eligible for FSM) (-0.14 to 0.12) 

Maths (cohort B & -0.09 n/a 
ever eligible for FSM) (-0.23 to 0.05) 0.21 -1 months 1,860 ££EF£EF 

Reading (cohort B & -0.04 n/a 
ever eligible for FSM) (-0.19 to 0.11) 0.61 0 months 1,791 EELEE 
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Introduction 


The Learner Response System (LRS) intervention, devised by Edge Hill University, provided Year 5 
and Year 6 primary school pupils in England (aged 9-11 years) with hand-held electronic clicker devices 
(ActivExpression, by Promethean). The handsets are supported by the software ActivInspire which also 
runs a Promethean interactive whiteboard. The handsets were to be used in lessons during the normal 
school day as a means of providing frequent, immediate feedback. Pupils receive individual feedback 
on their handsets and teachers receive it in aggregated form, on their computer, for display on the class 
whiteboard if they choose. The study assessed two cohorts of pupils: those who used the handsets in 
lessons from October—May during Year 6 (2014/2015 academic year) and secondly, those who used 
the handsets for the whole of their Year 5 as well as October—May during Year 6 (2014/2015 and 
2015/2016 academic years). 


Aconcise description of the intervention can be found in Box 1. The key components include the training 
for teachers and the use of the handsets with pupils. 


Programme resources and training 


Training for teachers 


Prior to the start of the intervention, an event was held for the headteachers of the intervention schools; 
this highlighted the functionality of the handsets, described previous evidence around their use, and 
explained the expectations of the evaluation. Subsequently, all Year 5 and Year 6 teachers from schools 
delivering the intervention were provided with training developed and delivered primarily by the LRS 
project team at Edge Hill University (specifically, the project lead and project Head of Training and 
Support). Others provided occasional training support as required, for example a specialist independent 
training consultant and a teacher with previous experience in using the LRS. The schools were 
organised into ten clusters, loosely arranged by geography, in which to receive the training. 


An initial two-day introductory training event was run at Edge Hill University in June and July 2014 for 
each cluster. Teachers who would be teaching Year 5 and 6 classes in the following school year (the 
first year of project delivery) were invited to attend. Schools were also encouraged to send their ICT 
lead to at least part of this training, although only a minority did so. The developers offered to pay the 
schools half of costs for supply cover for attendance at this, and subsequent, training. This event 
introduced teachers to the LRS equipment and how to set it up in their school environment, the theory 
behind the intervention, methods for basic usage (by teachers, and for teaching children how to use the 
handsets), the training and support available, and the intervention team’s expectations of the 
participating teachers and the evaluation. 


Five half-termly one-day training sessions were run, for each of the ten clusters of schools, during the 
first year of the intervention (approximately September and November 2014; January, February, and 
April 2015). Each of these was hosted by a school in the cluster and all Year 5 and Year 6 teachers 
from the cluster schools were expected to attend. Other members of staff who wanted more information 
or training (such as TAs and senior management team members) were also welcome to attend. These 
training days generally included some or all of the following: a review of progress in each school, 
presentations from schools showcasing work with the LRS, training on additional features of the system, 
and time for teachers to explore and experiment with resources with support from the Edge Hill team. 


In the second year, those teachers who were new to the project (for example, those new to teaching 
Year 6 in the school) were offered a full day’s introductory training (in June 2015) and a half-day visit— 
in school, from the Edgehill Head of Training and Support—in September 2015. They were expected 
to be released from teaching duties for this visit. These second-year teachers were then offered four 
half-termly one-day training sessions. Teachers new to the project in the second year were offered less 
training overall than those in the first year as the IT system was already operational in the school, 
reducing the need for training in that area. Continuing teachers were offered a ‘maintenance’ visit to 
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check the functioning of systems at the start of their second year on the study and training once a term 
for the first 2 terms (2 one-day sessions). Both first and second year teachers were invited to the end- 
of-project one-day conference. The training sessions covered the use of the system and the creation of 
resources such as flipcharts (the name given to a file within Activinspire). The Edge Hill project team 
also provided a staff member to be on-call to deal with problems faced by schools. 


Use of the handsets in classrooms 


The project supplied a handset for each Year 5 and Year 6 pupil in the schools allocated to the 
intervention. The handsets are recommended to be used at any point within lessons, in the following 
ways: 


e adhoc and pre-prepared questions—where a teacher asks a question (either ad hoc or pre- 
prepared) and pupils respond on the handsets (aggregated results can be displayed on the 
whiteboard immediately); 


e self-paced questions—where questions are pre-prepared and are sent to individual handsets. 


In the latter case, pupils work through the questions at their own pace. The system can be programmed 
by the teacher in a range of ways, for example, to tell the children if their answer is correct or incorrect 
and provide the correct answer (or not), or to ‘ask’ children to have one or more further attempt at an 
incorrect question, with hints and tips provided to support them. The system can be set to provide 
differentiation in the questions so that higher achieving pupils are automatically moved to more difficult 
questions. At the time the children are working on self-paced tasks, a results timeline is available to the 
teacher so he/she can see who is getting questions right or wrong, how long it is taking them, and 
whether or not there are periods of inactivity. This allows the teacher to intervene immediately when a 
pupil appears not to understand the concept being taught. Results of self-paced activities can be saved 
and downloaded, by a teacher, into an Excel spreadsheet for recording and analysis. Results can also 
be copied into a programme, called Inside, which is intended to provide a quick and easy way of seeing 
how many questions each individual child is getting correct or incorrect. Further information can be 
accessed from the developers of ActivExpression.' 


The intervention required teachers to use the handsets in a minimum core set of three lessons each 
week, to be spread over maths and English lessons. They were encouraged to use the handsets in 
other, more advanced and creative ways, such as using them in small groups or for other subjects. 
They were expected to upload their lesson plan resources to a dedicated Edge Hill website on a weekly 
basis that showed that the handsets had been used within their class. 


The LRS approach is based on the premise that timely and focused feedback is a powerful tool to 
facilitate learning. The overall objective of the LRS intervention is to improve the pace and quality of 
feedback and associated intervention by teachers. Feedback is conceived to be two-way; learners can 
respond to questions and instantly be told if they have made a mistake, and receive suggestions as to 
how they might rectify that mistake. In addition, teachers receive feedback on how each learner is 
progressing—for example, how long they are taking with each question, how many attempts they need, 
and where they are going wrong. This is meant to provide information to the teacher on which elements 
of a topic pupils understand or find challenging so that they can adjust their approach and feedback 
accordingly within the lesson and target support at struggling pupils. Figure 1 shows how the Edge Hill 
University developers perceive the inputs, change mechanism, and impacts of the programme. 


' LEB Partnership Ltd, (2013) ‘Questions for Learning (QfL) For users of ActivExpression 2 and Activinspire’, Lancashire: LEB 
Partnership Ltd, https://inside.learningclip.co.uk/Downloads/1/Questions_for_Learningv3_1.pdf 
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Figure 1. Learner Response System Logic Model, based on developers’ views 
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Box 1: LRS Intervention Description: TIDieR checklist 


Ti 
2. 


10. 


11. 


12. 


TIDieR checklist 


Brief name: Learner Response System 

Why—rationale, theory and/or goal of essential elements of the intervention: providing 
hand held feedback devices (ActivExpression, by Promethean) for use in classroom lessons 
would improve the children’s performance on maths and reading standard assessment tests 
(Key Stage 2 SATS). 

Who—recipients of the intervention: primary school pupils in Years 5 and 6 (age 9-11) in 
England. 

What—physical or informational materials used in the intervention: ActivExpression 
handsets (by Promethean) for every pupil, system software for schools, and core lesson 
resources for teachers to use or build upon. Teachers had the option of using either a 
laptop, PC, or tablet computer for utilising the system. 

What—procedures, activities, and processes used in the intervention: teachers use 
lesson plans provided by the developers or create bespoke lesson plans utilising the 
handheld devices so that children can, for example, test their learning via self-paced 
exercises, or provide feedback about lesson learning to the teacher. 

Who—intervention providers/implementers: the intervention training and curriculum was 
provided by a team at Edge Hill University. The delivery of the intervention was carried out 
by primary school teachers (usually the normal class teacher). 

How—mode of delivery: self-paced and whole-class lessons. 

Where—location of the intervention within primary school classrooms in the Northwest 
and West Yorkshire regions of England. 

When and how much—duration and dosage of the intervention: for one cohort of 
pupils, this took place during the 2014/15 academic year (25 weeks); for a second cohort 
this involved both the 2014/15 and 2015/6 academic years (32 weeks + 26 weeks). The 
handsets were expected to be used a minimum of three times a week during classroom 
lessons. 

Tailoring—adaptation of the intervention: teachers were provided with some bespoke 
classroom resources that they could use, but were encouraged in training to devise their 
own resources. 

Modifications: some schools used teaching assistants or an alternate teacher to lead the 
lessons that used the handheld devices. 

How well implemented (planned): researchers captured implementation data on 
intervention delivery via monitoring data and a teacher survey, as well as case study 
interviews and observations. 


Background evidence 


This study provides an opportunity to test the effectiveness, acceptability, and fidelity of a classroom 
based programme utilising electronic hand held feedback devices, also known as ‘learner response 


systems’. 


The most relevant, rigorous evidence for the use of such systems comes from two previous studies of 
the use of an electronic learner response system (called the Questions for Learning (QfL) strategy). 
These were conducted by the Institute for Effective Education (IEE) at the University of York. The initial 
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study, conducted in 2011, was a randomised evaluation involving seven U.K. primary schools.? This 
aimed to determine if technology-supported self-paced learning and formative feedback increased pupil 
learning in maths. The intervention produced positive effects on maths achievement (effect: +0.39, p > 
0.001). This promising finding led to a second study which aimed to determine if the approach could be 
successfully used in the teaching of grammar. The second study ran in 2012 and was a larger cluster 
randomised controlled trial involving 42 primary schools in the U.K.2 The schools were randomly 
assigned to use electronic handsets in the teaching of grammar in Year 5 classes, or to continue with 
ordinary grammar teaching. An additional 16 schools, in which the teachers had previous experience 
of using electronic handsets, took part as a supplementary comparison group. The intervention was 
delivered over a 12-week period, with a recommendation to use the handsets at least three times a 
week. This evaluation found positive effects on grammar but not on writing. The magnitude of the effect 
(0.16; p > 0.001) was enhanced for those who used the handsets at least three times a week (0.26; p 
> 0.01). The study suggested that average and low-achieving pupils’ learning of grammar concepts was 
particularly supported by the intervention. Acceptability by teachers and pupils was high. Improved 
formative assessment through the use of this technology was believed to be key to the improved pupil 
achievement in grammar. The authors recommended its use for other pupils and for other subjects. 
They concluded that if these results sustained over a school year, pupils who were low or average 
achievers would progress by an additional three to four months. 


Evaluation objectives 


The primary objective of the evaluation was to answer the research question: What is the effect of the 
LRS on children’s achievement in maths and Reading? 


The evaluation aimed to answer the following additional questions about the impact of the LRS on 
academic achievement: 


1. What is the effect of the LRS on children’s achievement in sub areas of maths and English 
literacy? 

2. Do treatment effects of the LRS differ across certain demographic groups (e.g. boys versus 
girls, those with/without free-school meal eligibility)? 

3. Are there heterogeneous treatment effects across the academic achievement distribution? 


Additionally, the evaluation aimed to explore a number of issues relating to the implementation and 
potential sustainability of the LRS programme. These included: 


1. Following LRS training, how feasible and acceptable is it for primary school classroom 
teachers to use the LRS in numeracy and/or literacy teaching? 

2. How does implementation of the LRS vary between teachers and schools? How much does it 
vary between subjects? 


3. What level of usage is achieved? 


e How feasible and acceptable is it for primary school pupils to use the LRS and how do they 
respond to the ensuing teacher feedback response in these classes? How do the children 
describe the benefits to learning of using the LRS? Do different sub groups experience it 
differently (e.g. low, medium and high attainment pupils)? 

4. What are staff perceptions of the impact of the intervention after 1 and 2 years on the quality 
of feedback pupils receive and subsequent impact on outcomes in literacy and numeracy? 
How do staff perceive it affects different sub groups (e.g. low, medium and high attainment 
pupils)? What are teachers’ perceptions of facilitators and barriers to impact? 


2 Sheard, M. and Chambers, B. (2011) ‘Self-paced learning: Effective technology supported formative assessment. Report on 
achievement findings’, York: The University of York. 

3 Sheard, M., Chambers, B. and Elliot, L. (2012) ‘Effects of technology enhanced formative assessment on achievement in 
primary grammar’, York: The University of York. 
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e How does feedback from teachers vary with use of the LRS devices? 


5. What are pupils’ perceptions of the impact of the intervention after one and two years on the 
quality and immediacy of feedback they receive and subsequent impact on outcomes in 
literacy and numeracy? What are pupils’ perceptions of facilitators and barriers to impact? 


Ethical review 
The project was submitted for review to the Institute of Education’s Faculty Research Ethics Committee 


and was granted ethical approval (263641 06/201 6/11/65). 


There were three layers of consent in this evaluation. First, school-level consent was obtained from 
head teachers for agreement to take part in the trial, to randomisation, and to access pupils’ data from 
the National Pupil Database (NPD). This school-level consent was followed by the distribution of an 
information letter about testing to parents, with an opt-out consent slip for parents to return if they did 
not wish to have their child participate in the testing or allow their child’s NPD test data to be used (see 
Appendix A for the Memorandum of Understanding for schools and Appendix B for the consent form 
for parents). The third layer of consent took place in case study schools where parents were given the 
opportunity to opt out of children’s participation in focus groups and teachers gave consent to interviews. 


Project team 


Edge Hill University, Learner Response System Programme Team: 
Janice Jackson, programme manager 

Pat May, Head of training and support 

Andy Done, teaching consultant and classroom resource developer 
Jon Chambers, training consultant 


Roger Gwinnett, programme initiator at Edge Hill 
Evaluation team: 


Meg Wiggins, principal investigator—oversaw impact and process evaluations, conducted process 
evaluation analyses and co-wrote the study report. 


Mary Sawtell led the process evaluation—conducted fieldwork and process evaluation analyses, and 
co-wrote the study report. 


Dr John Jerrim—designed the impact study, randomised the schools, conducted the analysis of the 
NPD data, and co-wrote the study report. 


We acknowledge the contribution of Dr Eleanor Hargreaves in the design of the process evaluation. 
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Methods 


Trial design 


The evaluation was a clustered randomised control trial, with randomisation at the school level. The 
evaluation team considered randomisation of the intervention (a) at the school level and (b) at the class 
level. Given the whole-class nature of the intervention, randomisation at school level was the preferred 
option. This was chosen to avoid possible contamination across classes, where there would be the 
potential for the devices to be shared with teachers in control classrooms, if intervention teachers 
thought it was working well. Additionally, by having school level randomisation, this eased practical 
issues within the schools, such as ensuring pupils in the same year groups all received the same 
treatment (limiting concerns about parental opposition), and fit in with the common practice of year 
group planning across classes. 


The control condition used was ‘business as usual’ within schools. Note that schools were designated 
as controls for two years—covering when the younger year group began Year 5 (September 2014) until 
they had left the school at the end of Year 6 (June 2016). This was done to avoid potential difficulties 
with contamination. The control sites were offered a half day of training and one classroom set of 
handsets in September 2016. 


This impact study was combined with an integrated process evaluation that had the scope to provide 
valuable insight into why the intervention has, or has not, had the intended impact on attainment, as 
well as explaining the variation in use of the LRS by different teachers. 


Participant selection 


The Institute of Education (loE) and Edge Hill University teams first identified specific Local Authorities 
(LAs) in the North West and West Yorkshire regions of England where Edge Hill had capacity to deliver 
the intervention. The LAs selected were: 


e Blackburn e StHelens 
e Blackpool e Stockport 
e ~=Bolton e Tameside 
e Bury e §=6 Trafford 

e Cheshire East e Warrington 
e Cheshire West e Wigan 

e ~=Halton e §=Wirral 

e Knowsley e Bradford 

e Lancashire e Leeds 

e Liverpool e Calderdale 
e Manchester e = Kirklees 

e Oldham e Wakefield 
e Rochdale e Barnsley 

e Salford e Staffordshire 
e Sefton 


The loE evaluation team then produced a list of all primary schools within these geographic regions. 
Private schools and schools where Edge Hill University already operated a similar system to that being 
trialled were excluded. The population of interest was further restricted to schools with a high intake of 
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disadvantaged pupils, based upon the percentage of children receiving free school meals as recorded 
in the 2011 school performance tables. (After reviewing the pool of schools, this was set to be at least 
55% of KS2 pupils who have ever been eligible for FSM, to ensure a wide enough pool for recruitment.) 
Thus the population of interest was defined as Year 5 and Year 6 state school pupils in schools within 
the selected geographic regions who attended a primary school with a high proportion of disadvantaged 
pupils. 


This final list of schools produced by the loE contained 533 schools that acted as the sampling frame 
(however, one of these 533 schools shut down meaning there were actually 532 for the project team to 
recruit from). Edge Hill University were then asked to recruit 100 schools from this list by the second 
week of May 2014. To recruit schools, Edge Hill University sent all schools an information pack and 
expression of interest form. Those that agreed to take part in the trial returned a headteacher consent 
form confirming participation in the study and allowing access to the NPD data for the school 
(Memorandum of Understanding, see Appendix A). Ninety-two schools were recruited to the trial by the 
initial target date. A further five schools were recruited a week beyond this deadline and were also 
included in the trial, bringing the total to 97. 


Outcome measures 


The primary outcomes were KS2 maths and English SATs test scores. These are national examinations 
children in England sit at the end of primary school (when pupils are typically age 10 or 11). These 
scores are a reliable, externally valid measure that is a strong predictor of children’s later educational 
outcomes. These are also ‘high stakes’ tests for schools in that schools are ranked in publicly available 
league tables by their pupils’ performance. These tests are not specific to the LRS intervention and are 
marked blind to treatment. These outcomes were pre-specified as part of the evaluation protocol. 


In December 2016, however, the EEF felt that English as an outcome was no longer appropriate given 
the fact that from 2016 writing would be assessed by teachers using one of three possible categorical 
descriptions. Therefore, it was decided that KS2 Reading would be used as a primary outcome as this 
is still marked blind by assessors and produces a raw or scaled score. As a secondary outcome, the 
KS2 Spelling, Grammar and Punctuation test will also be used. This is also marked blind by assessors 
and produces a raw or scaled score. This is a deviation from protocol. 


The outcome measures used in analysis varied between the two cohorts. The one-year intervention 
cohort took their exams in May 2015, and the NPD variables ks2_mattotmrk (primary maths outcome) 
and ks2_readmrk (primary reading outcome) were used to assess the outcome. For the two-year 
intervention cohort, the exams were taken in May 2016, and the NPD variables used were ks2matscore 
(primary maths outcome) and ks2_readscore (primary reading outcome). Secondary outcomes include 
the breakdown of children’s maths test scores into performance on the three separate KS2 SATS maths 
papers (two on reasoning and one on mental maths) and, as mentioned above, scores on the English 
grammar paper. 


Fidelity measure 


Monitoring data regarding attendance at training days was collected by the developer team at Edge Hill 
University and was shared on an ongoing basis with the evaluation team. Intervention teachers were 
also expected to upload evidence of use (for example resources, such as lesson plans) on a weekly 
basis to the Edge Hill team to ensure that they were completing the expected three lessons per week 
using the handsets. The Edge Hill team was in regular contact regarding this resource monitoring— 
chasing missing data, noting periods of low use, and helping schools problem-solve around challenges. 
A database which logged the number of resources submitted each week by each intervention teacher 
was supplied to the evaluation team at the end of each intervention year. For each teacher in the 
intervention, a summary of the type of resources submitted, and subjects covered by them, was also 
provided by Edge Hill on a yearly basis to the evaluation team. 
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Two fidelity measures were developed for use in the analysis of pupil attainment data, one for use with 
maths attainment analysis and the other for English attainment. These were applied to each cohort 
within each school and provided a ‘high’, ‘medium’, and ‘low’ categorisation of the intervention fidelity 
for the particular subject area, which focused on the intervention dose received. Following discussions 
with the developer team after the intervention was complete in 2016, the evaluation team created a 
metric that used an assessment of (1) monitoring data provided by the developer team at Edge Hill 
University which recorded the number of resources submitted by teachers teaching a particular cohort 
and the subject they related to and (2) teacher survey data about intervention use (see ‘process 
methods’ section below). This was calculated in April 2017, when process data analysis was complete. 
For the cohort which had two years of intervention delivery, assessment of fidelity was calculated for 
each of the two years and then averaged to form a single categorisation that could be added in the 
statistical analysis. 


Baseline test 


Children’s KS1 assessment scores, accessed from the NPD, were used to measure children’s 
academic achievement prior to the LRS intervention. These are based upon teacher assessments of 
pupils at age seven and thus before schools were randomly assigned to treatment or control groups. 
Indeed, at the point these baseline assessments were conducted, teachers would have been unaware 
that the LRS trial would take place. These baseline assessments are used to (1) investigate balance 
between treatment and control groups in terms or prior attainment and (2), when controlled for in 
statistical analysis, to increase power and reduce confounding. 


Sample size 


The evaluation team regarded 100 schools as the minimum necessary to detect an effect of 
approximately 0.20 of a standard deviation in maths test scores. This calculation assumed: 


e aninter-cluster correlation (ICC) of 0.15 at the school level (which is the approximate level of 
ICC across the population in England); 
e equal cluster sizes of 45 Year 6 pupils per school; 


e 40% of KS2 maths test scores variance explained by baseline covariates (that is, correlation 
between pre- and post-test of 0.63—see Table 5); and 


e 80% power for a 95% confidence interval. 


Below we consider how these assumptions compare to the actual data collected as part of this trial 
(Tables 5 and 6). 


Randomisation 


The trial was designed as a stratified, clustered randomised controlled trial with random allocation 
occurring at the school level. Schools were stratified by historical test scores drawn from the 2013 
school performance tables.‘ A principal components analysis (PCA)> was first conducted using the 
following four school-level achievement variables: 


e percentage of pupils reaching level 5 in KS2 maths in 2013; 
e percentage of pupils reaching level 5 in KS2 reading in 2013; 


4 Note the 2011 school performance tables were used to define the sampling frame (i.e. which schools Edge Hill 
University were asked to recruit into the trial). Data from the 2013 school performance tables is then used to stratify 
the sample. The 2013 data has been used to ensure that the most up-to-date information available on school 
performance is used to stratify the sample (under the assumption that more recent data is likely to provide a better 
guide to schools’ future KS2 performance). 

5 Stata.com (2017) Principle Component Analysis, https://www.stata.com/manuals13/mvpca.pdf 


Education Endowment Foundation 15 


Learner Response System 


e percentage of pupils reaching level 5 in KS2 Grammar, Punctuation and Spelling (GPS) in 
2013; and 


e average KS1 points score for the cohort of pupils sitting KS2 exams in 2013. 


The first principal component was then used to provide a continuous measure of school-level 
performance in 2013. The 92 schools recruited by the initial deadline were then divided into nine strata. 
A tenth stratum was then included to incorporate the five schools that were recruited after the deadline. 
The details and number of schools within each stratum can be found in Table 2. 


Table 2: Number of schools within each stratum 


Strata ID PCA achievement rank Schools per strata 
1 1st-10th 10 
2 11th—20th 10 
3 21st-30th 10 
4 31st-40th 10 
5 41st-52nd 12 
6 53rd—62nd 10 
7 63rd—72nd 10 
8 73rd—82nd 10 
9 83rd—92nd 10 
10 Late recruitment 5 


Within each stratum, a random number was drawn from a uniform distribution. The schools in the bottom 
half of the random draw distribution, within each stratum, were assigned to the control group. Schools 
with a number in the top half of the random draw distribution, within each stratum, were assigned to 
treatment.® The Stata statistical package was used to generate all random numbers, with a random 
seed chosen and set a priori to ensure the random allocation sequence is replicable. Note that schools 
in stratum 1—9 were randomly assigned to treatment and control groups on 7 May 2014 by Dr John 
Jerrim at loE. Schools in stratum 10 were randomised on 16 May 2014. 


Analysis 


Analysis of whether the intervention was effective or not was based upon the following OLS regression 
model using an intent-to-treat approach (that is, all pupils and schools were included in the analysis in 
their initially randomly assigned group, regardless of whether they took part in the intervention or not): 


Yio" = a+ B.Treat; + y.¥y" +6.Sy + &y (1) 


Where: 


6 For the five schools in the late recruitment strata, those with the lowest two random numbers were assigned to 
control and the schools with the highest three numbers assigned to treatment. 
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Vie = children’s KS1 score in maths or reading—whichever appropriate for outcome model (entered 
as a set of dummy variables).” 


yjes* = children’s KS2 score in maths or reading. 


S = a set of dummy variables reflecting the stratum each pupil/school was assigned to during the 
randomisation. 


Treat = a variable indicating whether the child was enrolled in a treatment or control school. 
€ = error term (with children clustered within school). 

i = child i. 

j = school j. 


To account for the clustering of pupils within schools, a Huber-White adjustment was made to the 
estimated standard errors (i.e. standard errors are clustered at the school level).8% The coefficient of 
interest from equation 1 is 8. This is the estimated impact of the LRS intervention. Cohen’s d was used 
to calculate the effect using the pooled standard deviation. 


After our main analysis, we re-estimated model 1 separately (1) for boys and girls, (2) for pupils ever 
eligible for FSM, and (3) using quantile regression to investigate differences in treatment effects across 
the achievement distribution. 


Changes from study protocol 


The following changes from the protocol were made in order to comply with certain aspects of the EEF’s 
policy on analysis: 


e Covariates included in the analysis model: the only covariates included in the analysis model 
below are (a) KS1 (pre-test) scores in the same domain as the outcome variable and (b) the 
stratification variable. The protocol pre-specified that OLS regression would be used, but 
without listing the relevant covariates. 


e Changes to KS2 test scores: it was initially intended to include all pupils across Year 5 and 
Year 6 in asingle analysis model, and include a dummy variable to investigate the impact of a 
one-year or two-year intervention against the control condition. However, KS2 tests changed 
across the 2014/2015 and 2015/2016 cohorts. A decision was therefore made to split the 
cohorts out (the former denoted ‘one year of intervention cohort’ or ‘Cohort A’ and the latter 
‘two years of intervention cohort’ or ‘Cohort B’) with separate analyses conducted for both. 


Implementation and process evaluation 


The process evaluation was integral to the trial and had three key objectives: to assess the fidelity of 
delivery of the intervention, to answer questions related to the feasibility of the intervention, and to 
support understanding of the results of the impact evaluation. 


7 Missing dummy variables are used to ensure observations are not dropped where KS1 test scores are 
unavailable. 

8 Huber, P. (1967) ‘The behaviour of maximum likelihood estimates under nonstandard conditions’, presented at 
the Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Statistics, 
The Regents of the University of California, http://projecteuclid.org/euclid.bsmsp/1200512988 


8 White, H. (1980) ‘A heteroskedasticity-consistent covariance matrix estimator and a direct test for 
heteroskedasticity’, Econometrica, 48 (4), p. 817. 


Education Endowment Foundation 17 


Learner Response System 


Constructing a logic model 


A logic model was developed to clarify the assumptions of LRS experts on the theory of change of the 
programme and to provide a framework to support the evaluation including the assessment of fidelity 
and explanation of findings. 


The construction of a logic model was undertaken by the research team in Spring 2015 using a Delphi 
consultation exercise, designed to achieve consensus within a group of eight experts involved with 
developing the LRS. The consultation was carried out by email without individuals conferring or seeing 
the responses of others in the group. The first stage asked for views on what the different components 
of the logic model were at each stage of the causal pathway. The components submitted by the eight 
participants were consolidated by the research team. The participants were then asked to rank the 
listed components in order of importance; seven responded. The research team analysed the ranked 
lists and constructed a logic model that reflected the combined views of the experts. The final version 
of the logic model can be found in the intervention description section (see Figure 1). 


Pre-intervention data collection 


Observation of Edge Hill initial training for teachers 


The 49 intervention schools were organised into 10 clusters by the Edge Hill team, loosely based on 
geography. Initial training was provided to schools at a two-day event, run multiple times in June and 
July 2014 by the intervention team at Edge Hill. Two clusters attended each two-day event; subsequent 
training sessions were delivered to one cluster at a time. These sessions were aimed at teachers who 
would be delivering the intervention; IT ‘leads’ from the schools were also invited. One member of the 
research team carried out non-participatory observation at one of these two-day sessions. Free-form 
observation notes were taken by the researcher. 


Data collection in Year 1 of intervention 


Observations of Edge Hill training 


Five half-termly training sessions were run for each of the ten clusters of schools during the first year 
of the intervention (approximately September and November 2014, and January, February, and April 
2015). Year 5 and Year 6 teachers from intervention schools attended the training. A researcher 
observed two of the November 2014 sessions and one February 2015 session. These three 
observations were all of different clusters. The clusters observed were purposively selected to ensure 
a spread of geographical location, size of cluster, and levels of enthusiasm for, and competency with, 
the LRS (the latter was informed by previous observations and feedback from the Edge Hill team). In 
June 2015, the Edge Hill team held a (non-compulsory) conference for intervention teachers where they 
celebrated the first year of the intervention and shared good practice. A researcher attended and 
observed the plenary sessions and two workshops. 


At the September 2014 and February 2015 training, a short self-complete pro forma, developed by the 
research team, was provided to teachers in all clusters. At both time points, these aimed to capture 
‘keenness’ and frequency of use of the LRS by the teacher as well as their own assessment of whether 
they used the LRS at a basic, intermediate, or advanced level. Teachers were also asked to document 
any challenges they had experienced using the LRS. The February version also included a question on 
individuals’ reflections on the intervention to date. These were given out by Edge Hill staff with an 
envelope for staff to seal to maintain confidentiality. The sealed envelopes were handed to Edge Hill 
staff who returned them unopened to the research team. 


Case Studies—selection and data collection 


Six case study schools were purposively selected for intensive multi-method data collection over the 
two years of intervention delivery. The purposive selection was conducted mid-way through the first 
year of intervention delivery (Feb 2015) to ensure a range of the following: 
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e levels of enthusiasm for, and engagement with, the intervention (this was assessed by 
teacher feedback data and observation of the training events, and information supplied by 
Edge Hill team members about challenges encountered with (or by) certain schools was also 
taken into account); 


e spread across the different training group clusters; 

e teaching experience, age, and gender of teachers; 

e numbers of year classes in a school; and 

e school average attainment levels and school proportions of pupils ever eligible for FSM. 


Visits were made, by one evaluation team researcher, to all of the six case study schools in the spring 
term of Year 1. 


Observation of lessons in the case study schools 


Non-participatory observations were carried out in Year 5 and Year 6 lessons in case study schools to 
provide information on how the intervention was delivered (with a particular focus on fidelity), the 
acceptability by all stakeholders, and barriers and facilitators to delivery. The researcher completed a 
semi-structured pro forma during the observation. Immediately after the lesson the researcher had a 
brief discussion with the class teacher to clarify any issues arising from the observation. These 
observations were carried out in five of the six schools; the sixth school had only managed episodic 
delivery in the first year and on the day of the site visit the LRS was not being used. 


Focus groups with pupils in the case study schools 


At least one pupil focus group was run in each case study school during the site visits in the first year 
of the intervention. The aim was to gain pupils’ views on the acceptability and usefulness of the LRS. 
Teachers were asked, in an email in advance of the site visit, to invite a mix of pupils to take part 
including Year 5 and Year 6 pupils, boys and girls, and higher and lower achievers. Five to six children 
took part in each group. Teachers were asked to arrange a private space for these to take place so that 
children could be assured of confidentiality. The focus groups were digitally recorded with the 
participants’ permission. All the focus groups commenced with a small number of questions about 
acceptability and impact—children were asked to record their individual responses on an anonymous 
slip of paper given out at the start of the group so that their opinion could be gleaned without being 
swayed by group views. For example, they were asked to write their answer to the question, ‘What do 
you think of the pods?’ (the LRS handheld devices) by responding ‘really good’, ‘OK’, or ‘not good’. The 
children were then invited to discuss their responses and a range of issues related to the use of the 
pods. Notes were made shortly after the focus groups to capture the key points (see focus group 
schedule, Appendix C). The audio recording was used as a check where there was uncertainty or to 
extract a particular quote. 


Interviews with Year 5 and 6 teachers in case study schools 


Semi-structured interviews were conducted face-to-face during site visits. Interviewees included Year 
5 and Year 6 class teachers, headteachers or deputy heads, teaching assistants, and others such as 
SENCOs who were involved with the classes using the LRS. Teacher interviewees were purposively 
selected to represent a range of year groups taught, IT proficiency, teaching experience, and subject 
specialisms. The research team used relevant theory and learning from the fieldwork and data collection 
in the study prior to the case study visits (Such as training observations and pro formas) to inform the 
selection of types of interviewees. The school was emailed in advance to request that either specific 
named teachers, or types of teachers, could be released from teaching duties to carry out the short 
interview. An element of pragmatism was also required (i.e. who was available on the day). Consent to 
interview, and for digital recording, were obtained. These interviews provided detailed information about 
the ways the LRS was being used in the classroom, perceptions of usefulness, and views on training 
and support by the Edge Hill team (see Appendix D for interview schedule). As with the focus groups, 
notes were made during and after the interview; audio recordings provided back up if required. 
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Staff surveys at the end of Year 1 with all schools 


An online survey of all Year 5 and Year 6 teachers in intervention schools was carried out in June—July 
2015 when intervention delivery was nearly complete. In intervention schools, any other staff who had 
opted to do some training and delivery of the intervention, such as teaching assistants and 
headteachers, were also sent the questionnaire. A second online survey was developed for completion 
by headteachers (or a nominated deputy) in control schools, where they were asked about their Year 6 
teachers and pupils. For schools in both trial arms, questions included information about teachers 
(gender, years since qualification), the class—including the amount of support in the classroom, other 
interventions during the year aimed at raising maths and literacy attainment, and numbers of pupils with 
particular needs such as SEN and EAL. Intervention teachers were also asked questions on the 
acceptability, feasibility, and sustainability of the intervention and on their perceptions of impact (see 
Appendix E). Two email reminders were sent to all non-responders which offered the option of 
completion over the telephone. An additional paper version of the questionnaire was sent by post with 
an accompanying prepaid reply envelope to all remaining control headteachers who had not responded. 


Data collection in Year 2 of the intervention 


Case study school visits in Year 2 


In the spring term of the second year of the intervention, pupil focus groups were conducted in five case 
study schools and interviews were carried out with Year 6 teachers in all six schools. A few class 
observations were also conducted in three of the case study schools. The methods employed were 
very similar to those of Year 1 though perceptions of intervention impact and sustainability from both 
teachers and pupils were particularly sought in these Year 2 visits. Where possible, interviews were 
with teachers in their second year of the programme to allow assessment of how their use, and also 
perceptions, of the LRS had modified with additional use. In some instances interviewees had only 
started using the LRS in the current school year. For one case study school, a telephone interview with 
a teacher was conducted. On the basis of this interview it was decided that data saturation had been 
achieved and a subsequent site visit was not carried out. 


Observation at workshops at end of Year 2 celebration event 


In the final term of the second year of intervention, the Edge Hill team held a celebration conference to 
mark the end of the intervention period. As part of this day, multiple workshops were held where schools 
highlighted their use of the LRS. Two members of the research team attended and observed workshops 
and spoke to teachers. Free-form observation notes were taken regarding innovative use of the LRS 
and about challenges faced in using the system. 


Teacher survey at the end of Year 2 with intervention schools 


An online teacher survey was carried out in July 2016 with just the intervention schools (Year 6 classes 
only). This aimed to capture similar data to the 2015 intervention survey but with a greater focus on 
teacher views on range of use of the LRS and on sustainability (see Appendix F). Three email reminders 
were sent out to non-responders requesting they log in to the survey to complete it. Any remaining non- 
responders were sent an email containing five key questions from the survey—with a request to reply 
to the email with their responses. 


Headteacher survey at the end of Year 2 with control schools 


A short control school headteacher survey was also conducted in July 2016. Because of the low 
response rate in the previous year, to facilitate engagement with this, just three simple closed questions 
about their Year 6 classes during academic year 2015/2016 were sent out in the body of an email to 
the headteacher. These questions asked about any usage of electronic hand-held learner response 
devices, and additional input to improve maths or English attainment. Headteachers were asked to 
send their responses by reply email. Non-responders were sent two follow-up emails followed by a 
paper version of the three questions sent by post with an accompanying prepaid reply envelope. 
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Analysis 


Framework Analysis was used for the analysis of the qualitative data from interviews, focus groups, 
and observations. This involved constructing frameworks based on key themes that answered the main 
research questions. This method allowed the exploration of the data by both theme and respondent- 
type enabling identification of patterns and associations across themes and types of respondents. 


Descriptive statistical analyses of the staff surveys and monitoring data was carried out using SPSS 
V22. Chi Square tests were used to measure statistical significance. 


Table 3: Process evaluation data collection—methods and response rates 


Method 


Class teacher end of Year 1 survey, 
treatment schools 


Sample size 


116 teachers 
(from 47 schools) 


Response: 
falUTanley=y ae 1(-)) 


93 (80%) 
(from 44 schools) 


Class teacher end of Year 1 survey, 45 6 (138%) 
control schools 
Class teacher end of Year 2 survey 64 teachers 50 (78%) 


(includes short email version), 
treatment schools 


(from 43 schools) 


(from 40 schools) 


Headteacher end of Year 2 survey, 
control schools (via email) 


Treatment arm: observations, interviews, 


Kole U-Me] cele] ox) 


45 


TUT pa) eY-yecmexey ave [lei (-ve| 
Year 1 


21 (47%) 


IN TUT pal eY=)ccomexe) ave [b(ey(=1e| 
Year 2 


Observation of Edge Hill two-day initial 1 0 

training for treatment school teachers 

Observation of Edge Hill half termly training | 3 0 

events, treatment school teachers 

Class teacher pro forma at training sessions | 228 from 48 schools 0 
Observation of workshops at end of year 2 4 
celebration event 

Case study schools, pupil focus groups 7 (42 pupils) 5 (30 pupils) 
Case study schools, observation of lessons | 10 3 

Case study schools, staff interviews 22 9 
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The cost per pupil per year was calculated by the evaluation team using data provided by the delivery 
organisation. The evaluators asked a series of questions about costs which the delivery team supplied. 
Calculations were made based on the cost data provided related to training, handsets, materials, and 
ongoing support. The assumption was made that there were 30 pupils per class, as this is standard for 
schools in England. As per EEF costing guidance, initial set up costs were spread over three years and 
annual running costs for three years were added to this in order to calculate a per pupil per year cost 
averaged over 3 years. The additional costs of school cover for training days was not included in these 
costings: arrangements for cover differ greatly between schools so cover requirements are presented 


separately. 


Timeline 


Table 4: Timeline of evaluation 


Date 


Activity 


June 2011 and June 2012 | Baseline KS1 tests conducted 
Dec 2013—May 2014 Schools recruited into trial including agreement of data access for NPD 
May 2014 Treatment and control schools assigned 


May—July 2014 


September 2014 


September 2014— 
July 2015 


February—April 2015 


May-June 2015 
May 2015 
September 2015 
February 2016- 
March 2016 


May 2016 
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Intervention headteacher event 
Treatment teacher’s baseline two-day training 
Observation of one of these training sessions 


Parental information and opt-out consent sheet distributed 
Trial intervention begins in Years 5 and 6 
Development of logic model using Delphi technique 


One day training per each half term for all treatment teachers 


Researcher observation of training in October 2014 and Spring 2015 
Teachers complete fidelity monitoring sheet at training days 


Case study site visits for class observation 
Teacher interviews and pupil focus groups 


On-line survey of teacher (Years 5 and 6, intervention and control) 


Year 6 pupils (1 year of intervention) sit KS2 SATs (numeracy/literacy) 


Trial intervention continues for second year for previously Year 5 pupils, 
now in Year 6 


Case study site visits for class observation 
Teacher and headteacher interviews and pupil focus groups 


Year 6 pupils (2 years of intervention) sit KS2 SATs (numeracy/literacy) 
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Impact evaluation 


Participants 


Figures 2 and 3 below illustrates the flow of the participants who participated in the study, for the two 
primary outcomes: maths and reading. Out of the population of 532 eligible schools, 97 took part. Half 
of these were allocated to the intervention (49 schools) and half to the control group (48 schools). 
Although consent to access the NPD data was gained from all of the schools allocated to the 
intervention, three control schools refused to continue their participation and dropped out of the study 
(including the impact evaluation, as they refused access to NPD data). Additionally, for one school 
allocated to the treatment group, there were problems in collecting KS2 test scores data for pupils who 
received two years of the LRS intervention—cohort B. Consequently, data available for analysis came 
from 45 out of the 48 schools allocated to the control group. Additionally, there was data available for 
analysis from 49 treatment group schools from their classes who had one year of the intervention 
(cohort A) and from 48 treatment schools from their classes that had two years of the intervention 
(cohort B) (see Figures 2 and 3). There was no evidence of any contamination between the treatment 
and control groups. 
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Figure 2: Flow of participants—maths 


Eligible schools 
(school n = 532) 


Declined to participate or 
did not respond to invitation 
(school n = 435) 


Eligible pupil = 35,726 


Recruitment 


Randomised 
(schools, n = 97; pupils, n = ~6,572) 


Allocated to intervention Allocated to control 
49 Schools 48 schools 


Allocation 


~ Cohort A (1 yr ~ Cohort A (1 yr intervention) 1721 
intervention) 1492 pupils pupils 

~Cohort B (2 yrs ~Cohort B (2 yrs intervention) 
intervention) 1570 pupils 1789 pupils 


Follow up Follow up Lost to drop out= 
Cohort A 
49 schools 45 schools 3 schools dropped out 


Lost to drop out ~ 
Cohort A 


0 schools dropped out 
2 - " 152 pupils lost (53 school 
105 pupils lost (105 no UPN or Pupils: Pupils dropout; 99 no UPN or no match 


cauahiaiaia Cohort A: 1387 Cohort A: 1569 abolownp) 
Conor: Cohort B: 1454 Cohort B: 1700 Cohort B 


1 school dropped out 3 schools dropped out 


Follow-up 


116 pupils lost (58 school 89 pupils lost (68 school dropout; 
dropout; 58 no UPN or no 21 no UPN or no match at follow 
match at follow up) up) 


Not analysed Analysed: Analysed: Not analysed 
51 Cohort A 68 cohort A pupils 
pupils without 49 schools and 1,336 45 schools and 1,501 without KS2 maths 


KS2 maths pupils in cohort A. pupils in cohort A SCORES: ; 
scores. Plus 14 Cohort B pupils 


13 Cohort B pupils scores.without KS2 
without KS2 maths 48 schools and 1,441 45 schools and 1,686 athe 


scores. pupils in cohort B. pupils in cohort B. 


Analysis 


Note - Reasons for no UPN or match at follow-up could include, for example, moving or not being entered into 
SATS. 
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Figure 3: Flow of participants—reading 


Eligible schools (school 
n=532) 


a Te Declined to participate or 
Eligible pupil = 35,726 did not respond to invitation 


(school n=435) 


Recruitment 


Randomised 
schools= 97; pupils=~6572 


Allocated to intervention Allocated to control 
49 Schools 48 schools 


Allocation 


~ Cohort A (1 yr ~ Cohort A (1 yr intervention) 1721 


intervention) 1492 pupils pupils 
~Cohort B (2 yrs ~Cohort B (2 yrs intervention) 
intervention) 1570 pupils 1789 pupils 


Lost to drop out ~ Follow up Follow up Lost to drop out= 


Cohort A Cohort A 


49 schools 45 schools 


0 schools dropped out 3 schools dropped out 


105 pupils lost (105 no UPN or “, " 152 pupils lost (53 school 
no match at follow up*) Pupils: Pupils dropout; 99 no UPN or no match 


Cohort B Cohort A: 1387 Cohort A: 1569 at follow up) 
Cohort B: 1454 Cohort B: 1700 Cohort B 


3 schools dropped out 


Follow-up 


1 school dropped out 


116 pupils lost (58 school 

dropout; 58 no UPN or no 89 pupils lost (68 school dropout; 

match at follow up) 21 no UPN or no match at follow 
up) 


Not analysed Analysed: Analysed: Not analysed 
53 Cohort A 74 cohort A pupils 
pupils without 49 schools and 1,334 45 schools and 1,495 without KS2 reading 


KS2 reading pupils in cohort A. pupils in cohort A Bootes: : 
scores. Plus 73 Cohort B pupils 


68 Cohort B pupils scores.without KS2 
without KS2 48 schools and 1,386 45 schools and 1,627 reading scares 


reading scores. pupils in cohort B. pupils in cohort B. 


Analysis 


Tables 5 and 6 illustrate how the power calculation presented above change when the final pre/post- 
test correlation, ICC and participants numbers are used. Although the number of schools and pupils in 
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the analysis is slightly lower than anticipated, this has been offset (in terms of power) by a lower ICC 
than expected. The final minimum detectable effect size within this clustered RCT is 0.18 - slightly lower 
than the 0.20 initially anticipated. 


Power calculations—maths 


Table 5A: Cohort A (one year of intervention) 


N school 
nie Intervention: n= Correlation Minimum 
(oxo} alice} )) 
between pre- Alpha fo (=) ¢=Yeq ¢-] 0) (:) 
test and post- effect size 
N pupils test (1/1 B) =5)) 
(n = intervention; n = 
exe} al iae))) 
100 (50; 50) 
Protocol 0.630 0.150 0.8 0.05 0.2 


4,800 (2,400; 2,400) 


97 (49; 48) 
Randomisation 0.653 0.131 0.8 0.05 0.18 
3,213 (1,492; 1,721) 


94 (49; 45) 
Primary analysis 0.653 0.131 0.8 0.05 0.18 
2,837 (1,336; 1,501) 


Table 5B: Cohort B (two years of intervention) 


iN Yoda Tole) [3 
a ea biel Correlation Minimum 
between pre- Aloha fo (=) (=Yeq ¢-] 0) (:) 
test and post- P effect size 
N pupils test (V2) 5) 
(n = intervention; n = 
Coxe} al iae))) 
100 (50; 50 
Protocol ( ) 0.630 0.150 0.8 0.05 0.2 
4,800 (2,400; 2,400) 
97 (49; 48 
Randomisation ( ) 0.686 0.115 0.8 0.05 0.18 
3,359(1,570; 1,789) 
93 (48; 45 
Primary analysis ( 0.686 0.115 0.8 0.05 0.18 
3,127 (1,441; 1,686) 
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Power calculations—reading 


Table 6A: Cohort A (one year of intervention) 


N schools 
(OS apieenCn Correlation Minimum 
eontc)) between pre- fo (= (Tey t=] 0) (=) 
test and post- [ee power pipe effect size 
N pupils test (12) SS) 
(n = intervention; n = 
(oxey alte) )) 
100 (50; 50) 
Protocol 0.630 0.150 0.8 0.05 0.20 
4,800 (2,400; 2,400) 
97 (49; 48) 
Randomisation 0.595 0.105 0.8 0.05 0.18 
3,213 (1,492; 1,721) 
94 (49; 45) 
Primary analysis 0.595 0.105 0.8 0.05 0.18 
2,829 (1,334; 1,495) 


Table 6B: Cohort B (2 years of intervention) 


N schools 
me Spee ee Correlation Minimum 
between pre- fo (=) (=Yeq ¢=] 0) (=) 
test and post- effect size 
N pupils test (112) ==) 
(n = intervention; n = 
control) 
100 (50; 50 
Protocol ( ) 0.630 0.150 0.8 0.05 0.20 
4,800 (2,400; 2,400) 
97 (49; 48 
Randomisation ( ) 0.585 0.096 0.8 0.05 0.18 
3,359(1,570; 1,789) 
93 (48; 45 
Primary analysis ( ) 0.585 0.096 0.8 0.05 0.18 
3,013 (1,386; 1,627) 


Pupil characteristics 


Table 7 provides information about the schools in the sample. Overall, the schools were mainly 
Community schools (59%), although a substantial proportion were Faith schools (voluntary aided or 
controlled) (36%). They had a global rate of 54% of pupils having ever been eligible for FSM. There 
were differences between the intervention and control schools, especially in terms of Ofsted ratings and 
numbers with English as an additional language, however none of these differences were statistically 


significant. 
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Table 7: School level characteristics, by trial arm 


WYF-Ta tle) (=) Taltclaccyaliceya mere svete) fy eto} a} (ce) Moye aVore) [> 


Sod solo) ei (=\'/-) | n/N 


(Cen celdater-1)) Geet) Percentage n/N (missing) Percentage 


Type of school@ 

Community 31/49 
Faith school@ 17//49 
Foundation 

Academy? 


Ofsted rating® 
Outstanding 

Good 

Requires improvement 


Inadequate 


School size 
1 form entry 
2 form entry 
3 form entry 


School-level Mean Mean 
(continuous) Percentage =a 


% English as ' , 
additional language® ey 14.7% Mee [ee 21.2% 


4 Dfe Data 2013/2014. ‘Faith school’ includes Voluntary Aided and Voluntary controlled schools; ‘Academy’ refers 
to any school that has academy status (whether a community or faith school) and also includes academy 
converters. 

> Ofsted rating at 1 May 2014. 

° Missing data because figure suppressed by DfE to protect anonymity. 


n (missing) n (missing) 


Tables 8 and 9 compare the intervention and control groups in terms of pre-intervention characteristics, 
all of which are measured at the individual pupil level. (These figures refer to only the pupils in the final 
analysis sample—as detailed at the bottom of Figure 2.) 


For cohort A, the treatment and control groups are well balanced in terms of gender (49% male in both 
the treatment and control groups) and eligibility for FSM (86% in both groups). However, there is a clear 
difference in terms of KS1 test scores. Specifically, for all three subject areas (maths, reading, and 
writing) the treatment group had more pupils working at level 2a and level 3. This is then reflected in 
the bottom row of Table 8, with the intervention group having a higher KS1 average points score than 
the control group (14.3 versus 13.8). This difference is equivalent to an effect size of around 0.14 
standard deviations (not presented) in baseline test scores (in favour of the control group). Hence for 
cohort A there is evidence of some imbalance in pre-intervention achievement between the intervention 
and control groups. 


For cohort B, there seems to be evidence of better balance in pre-intervention characteristics between 
treatment and control groups. As for cohort A, there is a similar proportion of boys and pupils currently 
eligible for FSM in the treatment and control groups. However, KS1 (pre-intervention test scores) now 
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seem to also be more similar across the two arms, with only a slight advantage to the control group. 
For instance, in KS1 maths, 26.2% of the intervention group achieved level 2a and 13.5% level 3. This 
compares to 23.2% and 13.2% in the control group, respectively. Similar findings hold for the KS1 
reading and writing domains as well. The overall difference in average KS1 scores between pupils in 
the treatment and control groups for cohort B is therefore small, equivalent to an effect size of 0.02 (not 
presented). Hence our interpretation is that, for cohort B, this trial is well balanced in terms of pre- 
intervention characteristics. 


Table 8: Baseline comparison for cohort A (one year of intervention) 


WE Tarlo) (=) Taiccyavc-valdcelame| ele e) Ofoy alice) Me [cole] e) 


| Pupil-level (categorical) BICWicl im (er-1k-Xe Le) a (er-1)) A ___sé~Percentage — n/N n/N (missing) Percentage _ Percentage 


Currently eligible for FSM 35.9% 35.5% 
Sex, male 49.3% 49.1% 


KS1 maths 


Below Level 1 
Level 1 

Level 2c 
Level 2b 
Level 2a 
Level 3 


KS1 reading 
Below Level 1 
Level 1 

Level 2c 
Level 2b 
Level 2a 


Level 3 


KS1 writing 


Below Level 1 
Level 1 

Level 2c 
Level 2b 
Level 2a 
Level 3 

KS1 APS 
Mean 


Standard deviation 
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Table 9: Baseline comparison for cohort B (two years of intervention) 


AYEVar-1+)(-3 Tai(claV{-1al cela me) cele] e) | fey alige) Me [cele] e) 


| Pupil-level (categorical) Percentage | n/N(missing) Percentage 


34.9% 575 33.9% 
48.4% 848 49.9% 
KS1 maths 
Below Level 1 
Level 1 
Level 2c 
Level 2b 
Level 2a 
Level 3 
KS1 reading 
Below Level 1 
Level 1 
Level 2c 
Level 2b 
Level 2a 
Level 3 
KS1 writing 
Below Level 1 
Level 1 
Level 2c 
Level 2b 
Level 2a 
Level 3 
KS1 APS 
Mean 


Standard deviation 


Outcomes and analysis 


Primary outcomes 


Table 10 presents the results for cohort A, those who received 1 year of intervention. For both reading 
and maths, the mean score is higher for the treatment group than the control group. These results, 
however, ignore the imbalance that was observed in baseline (KS1) characteristics between these two 
groups. Once this has been adjusted for in the analysis model the effect size becomes small in terms 
of magnitude (i.e. 0.00) and never approaches statistical significance at conventional thresholds. 
Specifically, after receiving the Learner Response System intervention for one academic year, the 
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intervention group essentially performed the same on their KS2 maths and reading tests as the control 
group (effect size of 0, as estimated using Cohen’s d). 


The analogous results are presented in Table 11 for cohort B, those who received 2 years of the 
intervention. Unlike cohort A, Cohort B was balanced on baseline KS1 characteristics: the mean KS2 
scores for cohort B were very similar for the control group and the treatment group in both reading and 
maths. The effect size in maths estimated from the OLS regression model is negative at -0.08, but not 
statistically significant. Specifically, the 95% confidence interval ranges from -0.21 to +0.05 in maths 
and -0.17 to +0.10 in reading. The key finding is therefore consistent across both cohorts; the LRS 
intervention had no impact upon pupils’ KS2 maths and reading test scores. 


Table 10: Primary outcome estimates for cohort A (one year of intervention) 


Raw means/n (%) Effect size 


Intervention 


| (fey alice) Me [cele] e) 


am tamricere(=)| 95% Cl 
Primary Mean Effect 
ColU) coxe) sa(- mim (atx) Lae ) Bi =I 2D) (missing) (62) size Lower Upper 
KS2 
1,336 68.7 67.3 

maths (156) (19.3) 1501 (220) (19.7) 2,837 0.00 -0.13 0.14 0.96 
score 
KS2 

. 1,334 29.6 28.7 
hale (158) (8.7) 1495 (226) (8.8) 2,829 0.00 0.12 0.13 0.96 


Table 11: Primary outcome estimates for cohort B (two years of intervention) 


Raw means/n (%) Effect size 


ai teyaV{claid(o)ale| cele] 0) fo} alice) Me] cele] e) 


1 OF | 
Primary a ——_=—— Effect 

ColU Kero) a at- Mam (LUT S-1 1010 ) MN (12) size Lower Upper 
KS2 maths 1,441 101.2 101.6 
eats (129) (7.9) 1686 (103) (8.0) 3,127 -0.08 -0.21 0.05 0.24 
KS2 

, 1,386 100.6 100.5 
reading (184) (7.9) 1627 (162) (8.1) 3,013 0.04 0.17 0.10 0.60 
score 

Sub-groups 


Tables 12-15 repeat our analysis for three pre-specified sub-groups: (1) pupils ever eligible for FSM, 
(2) boys, and (3) girls. Very similar results emerge as for the main results in both Cohorts A and B. 
Effect sizes from the intervention model are small for all groups in both KS2 subject areas, with the 
coefficient for the effect negative for pupils ever eligible for FSM and boys in cohort B. However, none 
of these estimates reach statistical significance at conventional thresholds. There is consequently no 
evidence that a one- or two-year dose of the intervention led to any improvement in KS2 scores for 
boys, girls, or pupils who have ever been eligible for FSM. 
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Table 12: Primary outcome results by sub-group. Cohort A (one year of intervention)—maths 


FSM 65.8 64.6 1,677 (778; 

cupits | 72 | og | 89 | a97 899) 0.01 -0.15 0.14 0.94 

Boys 676 (0) nee 764 (0) oF ee 0.02 | 0419 | 0.6 0.80 
67.8 65.9 1,397 (660; 

Girls 66010) agg) 737) gz 737) 0.01 | -0.17 | 014 | 0.87 


Note: ‘FSM pupils’ refers to pupils ever eligible for FSM. 


Table 13: Primary outcome results by sub-group. Cohort A (one year of intervention)—reading 


FSM 28.3 27.8 


cupits | 775 | gx | 2960 | ga) 1,671 0.01 -0.14 0.12 0.85 
28.8 28.1 

Boys 674(0) ig) 7580) gg) 1,432 0.01  -0.14 015 0.93 
30.5 29.2 

Girls 6600) ig) 7870) gay 1,397 0.00 -013 013 0.99 


Note: ‘FSM pupils’ refers to pupils ever eligible for FSM. 


Table 14: Primary outcome results by sub-group. Cohort B (two years of intervention)—maths 


FSM 99.9 100.5 1,860 (873; 


pupils 873 (0) (8.0) 987 (0) (8.0) 987) -0.09  -0.23 0.05 0.21 
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Boys 747 (0) a 841 (0) nn ene 015  -030 0.00 0.05 
Girls 694 (0) me 845 (0) oe eta 0.00 | -0.14 | 0.13 | 0.95 


Note: ‘FSM pupils’ refers to pupils ever eligible for FSM. 


Table 15: Primary outcome results by sub-group. Cohort B (two years of intervention)— 


reading 
Raw means/n (%) 


Talcclaccyaliceyal 


(Ofoy alice) me |cele]e) 


Effect size 


am lamanreye(=)| 95% Cl 
Group (missing (intervention; Lowe Uppe 
y control) r 
FSM 99.5 99.4 1,791 (836; 
pups 828) Ge) | 9550) Gg oss) 0.04  -0.19 0.11 0.61 
100.1 100.3 1,518 (706; 
Boys §706(0) 74) 8120) gy Pep 0.06  -0.21 0.10 0.48 
: 101.1 100.7 1,495 (680; 
Girls 680(0) gq 81500) ign ae -0.01 | -0.15 | 0.12 0.87 


Note: ‘FSM pupils’ refers to pupils ever eligible for FSM. 


Secondary outcomes 


Tables 16 and 17 proceed by considering the initial secondary outcomes, which refer to the sub- 


domains on the KS2 maths test. For cohort A, there is no evidence o 
paper B on the maths test, with effect sizes of 0.03 and 0.01 respective 
B, with a (non-significant) negative effect on paper 2 (-0.09) and paper 


f any effect in either paper A or 
ly. Similar results hold for cohort 
3 (-0.04). The coefficient for the 


estimated treatment effect in mental arithmetic is negative for both cohorts (-0.05 and -0.03). 


Table 16: Secondary outcomes—maths subdomains, cohort A (one year of intervention) 


Raw means/n (%) 


Intervention Control group 


ram iamanteye(=)| 


n 
Group (missin 


g) control) 


(intervention; 


Effect size 


95% Cl 


| o)'\ mm © o) 8) 
er er 
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1,336 27.1 1,501 263 2,837 (1336; 

Paper A (156) (7.9) (220) (8.0) 1501) 0.03 | -0.11 0.17 0.67 
1,336 286 1,501 28.0 2,837 (1336; 

Paper B (156) (8.0) (220) (8.1) 1501) 0.01  -0.14 0.16 0.88 

Mental 1,336 13.1 1,501 13.0 2,837 (1336; 

arithmetic (156) (4.9) | (220) (5.0) 1501) SNS POs O08 4) Ose 
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Table 17: Secondary outcomes—maths subdomains, cohort B (two years of intervention) 


Maths 1,407 20.6 1,645 21.2 


papers ea) | 9) | 4 | ea) 3,052 0.09 -0.23 0.06 0.24 
Maths 1,406 195 1,645 19.6 

papers Baa Wve ae Naeaaedl aes 3,051 0.04 -0.18 0.10 0.58 
Mental 1,407 28.8 1,645 29.0 e058 aoa asi was ae 


arithmetic (163) (8.9) | (144) — (8.6) 


Finally, Tables 18 and 19 consider the secondary outcome of a subdomain of the English SATs paper: 
the Grammar, Punctuation and Spelling test (GPS). For cohorts A and B, there is no evidence of any 
effect in the GPS test, with (non-significant) negative effect sizes of -0.01 and -0.03, respectively. 
Overall, we therefore find no evidence of any impact on either the primary or secondary outcomes for 
either cohort A or cohort B. 


Table 18: Secondary outcomes—English GPS test, cohort A ( one year of intervention) 


1,354 4.8 1,550 4.7 


GPs (138) (0.9) (220) — (0.9) 


2,904 -0.01 -0.12 0.11 0.92 


Table 19: Secondary outcomes—English GPS test, cohort B (two years of intervention) 


1,404 47.7 1,639 47.6 


ar (166) (13.6) (103) (12.9) 


3,043 -0.03 -0.15 | 0.10 | 0.70 
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On-treatment analysis 


Table 20 provides evidence of the association between the magnitude of the ‘dose’ of the intervention 
delivered within schools and pupils’ scores on their KS2 examinations. The ‘dose’ measure is based on 
the number of lessons the handsets were used in, as reported by the programme developers (Edge 
Hill) and reported in questionnaires sent to the evaluation team (see fidelity section in Process 
Evaluation findings for more details). Note that as schools have essentially ‘self-selected’ into these 
different dose categories, we cannot interpret these figures as causal effects. Rather, they illustrate the 
correlation between intervention dose and KS2 primary outcome measures. 


For cohort A, there was a negative (though not statistically significant) coefficient for schools that 
delivered a low dose of the LRS intervention to their pupils. This was true for both reading and maths. 
For schools in the medium dose category, the effect was small for both reading and maths and not 
statistically significant (effect sizes of 0.06 in maths and 0.03 in reading). Finally, schools that received 
a high dose of the LRS intervention did no better than the control group in maths (0.03 effect size). 
Although the effect size was large and positive for high-dose schools in reading (0.27 effect size) a 
great deal of caution is required in interpreting this result given the small number of pupils and schools 
who met the high dose threshold for use of the LRS in English (it includes just 49 pupils). Overall, the 
results from cohort A suggest that schools with low fidelity to the LRS did worse than control schools, 
while the picture is more positive for medium dose. Meanwhile schools that delivered the LRS with high 
fidelity may have seen more marked improvements in reading compared to control schools, but the 
small sample size makes these results more difficult to interpret with confidence. 


Table 20: Association between dose received and LRS impact, cohort A (one year of 
intervention) 


Maths 


Raw means/n (%) Effect size 


Dose 


Primary n(missing) Mean(SD) nin model Effect 95% Cl p-value 
outcome er | - 


| Wo) (-) am 6) ©) o=g 


i ls 2 1,501 (0) | 67.3 (19.7) 2,837 

Low dose 250(0) 66.7 (19.3) 2,837 0.20  -049 0.09 0.19 

Medium dose 680(0) 70.4 (19.3) 2,837 0.06 -0.10 0.21 0.50 

High dose 406(0) 67.3 (19.2) 2,837 0.03 -018 0.24 | 0.78 
Reading 

Control group 1,495 (0) 28.7 (8.8) 2,829 

(REF) 

Low dose 357(0) 29.3 (8.5) 2,829 0.10 | -0.27 | 0.07 | 0.26 

Medium dose 928(0) 29.6 (8.7) 2,829 0.03 | -0.11 | 017 | 0.70 

High dose 49(0) 33.1 (7.4) 2,829 0.27 0.13 0.40 0.00 
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@ REF refers to the reference group in the model. So the ‘effects’ reported are all in comparison to the control 
group. 


Table 21 provides analogous results for cohort B, with similar results emerging. In maths, the coefficient 
for the low and medium dose groups is negative, though not statistically significant. Likewise, the effect 
size for the high dose group is very small and insignificant (0.02). There is hence no evidence that even 
schools that chose to deliver a high dose of the LRS intervention did better than the control group on 
their KS2 examinations. Similar results hold for reading, though again with a positive (but insignificant) 
estimated effect for the high dose group (+0.10). Overall the evidence from across Tables 20 and 21 is 
that schools that delivered medium doses of the LRS intervention did not perform better on the primary 
outcome measures than the control group, and even for the high dose schools there is insufficient 
evidence to conclude that the intervention had a positive effect. 


Table 21: Association between dose received and LRS impact, cohort B (two years of 
intervention) 


Maths 


Raw means/n (%) Effect size 


ram iamanteye (=| 95% Cl 
Effect 
size 


Primary 


Mean (SD) 
Col V) Kero) ant) 


Woh (-) am 6) 0) of = g 


Control group 

(REF) 1,686 (0) | 101.6 (8.0) 3,127 

Low dose 344 (0) 101.0 (8.1) 3,127 -0.12 -0.31 0.07 0.21 
Medium dose 604 (0) 101.0 (7.7) 3,127 -0.13 -0.33 0.06 0.18 
High dose 493 (0) 101.4 (8.1) 3,127 0.02 -0.15 0.19 0.82 
Reading 

Control group 

(REF)® 1,627 (0) | 100.5 (8.1) 3,013 

Low dose 599 (0) 100.0 (8.1) 3,013 -0.11 -0.28 0.05 0.17 
Medium dose 482 (0) 100.8 (7.6) 3,013 -0.02 -0.23 0.18 0.84 
High dose 305 (0) 101.3 (7.7) 3,013 0.10 -0.12 0.33 0.36 


4 REF refers to the reference group in the model. So the "effects" reported are all in comparison to the control 
group. 


Analysis (quantile regression estimates) 


Although there is little evidence that the intervention has an impact on average, it is possible that it 
could have an impact at either the top or bottom end of the achievement distribution. In our study 
protocol, we specified that we would investigate this possibility by re-estimating our analysis model 
using quantile regression at each achievement decile. These results are presented in Figures 4 and 5. 
Overall, there is very little evidence of any substantial effect at any point of the achievement distribution. 
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The results for reading are zero throughout the distribution, suggesting the impact had no impact upon 
either high, low or average achievers. Likewise, there is no clear pattern for maths, with effect sizes 
always small (and often negative) in terms of magnitude. Overall, Figures 4 and 5 provide no evidence 
of heterogeneous impacts of the intervention across the KS2 achievement distribution. 


Figure 4: Quantile regression estimates of the association between the intervention and Key 
Stage 2 scores at each achievement decile, cohort A (one year of intervention) 


0.20 


0.10 
® 
N 
i?p) 
> 0.00 
2 
th 

-0.10 

— = Maths Reading 
-0.20 
10 20 30 40 50 60 70 80 90 


Percentile 


Note: Figures refer to conditional quantile regression estimates, controlling for KS1 scores in the subject and the 
stratification variable. 
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Figure 5: Quantile regression estimates of the association between the intervention and Key 
Stage 2 scores at each achievement decile, cohort B (two years of intervention) 


0.20 


0.10 
® 
N 
wn 
> 0.00 
& 
ti 
-0.10 7. \ 
=— — Maths Reading 
-0.20 
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Note: Figures refer to conditional quantile regression estimates, controlling for KS1 scores in the subject and the 
stratification variable. 


External validity 


Schools were not randomly selected into the trial. Rather, the evaluators compiled a list of 533 schools 
within the 29 local authorities who were eligible to participate in the trial (see methodology section on 
‘designing the sampling frame’). This list of schools was then given to the Edge Hill project team which 
recruited 97 schools to participate in the trial. Put another way, the project team achieved an 18% 
recruitment rate among the 532 eligible schools. In this section, we investigate how representative the 
trial participants across the 97 schools were compared to the population of pupils across all 532 eligible 
schools, and compared to state school pupils across the country as a whole. 


Table 22 considers whether pupils within the 94 participating schools (for wnom KS2 maths outcome 
data is available) have similar baseline (KS1) test scores to pupils in the population of 532 schools who 
were eligible to take part in the trial. (Figures for all state school pupils in England are also provided for 
context, though the trial has not been designed to generalise to the country as a whole).'° Cohorts A 
and B have been combined for the purpose of this table. 


The percentage of children in each KS1 performance level is very similar across the ‘trial participants’ 
and ‘eligible’ samples. A similar finding holds for the distribution of KS1 levels across each of the three 
subject areas; differences between trial participants and the eligible population is never more than one 
or two percentage points. Thus, despite the absence of random sampling, children who took part in the 
trial were very similar to the population of pupils they were meant to represent in terms of prior academic 
achievement. 


10 As expected, children who were eligible to participate in the trial were more likely to be eligible for FSM and have 
lower KS1 test scores than the state school population of England as a whole. 


Education Endowment Foundation 38 


Learner Response System 


Table 23 presents a similar comparison for a range of other observable characteristics. There are 
slightly fewer children with English as an Additional Language (EAL) amongst trial participants (19%) 
than in the eligible population (23%). The same holds true for FSM (35% versus 40%). On the other 
hand, the percentage of male pupils and eligibility for FSM was almost identical across the ‘trial 
participants’ and ‘eligible population’ groups. Overall, this reinforces the main message of Table 23, 
that the sample of trial participants is broadly representative of the population who were eligible to take 
part. 


Table 22: Comparison of Key Stage 1 test scores of trial participants to (1) the population of 
eligible pupils and (2) all state school pupils in England 


Trial participants All eligible pupils tare} E-lare| 


(%) (%) 7) 


Key Stage 1 maths 


Level 1 and below 13 14 8 
Level 2A 22 23 27 
Level 2B 29 30 27 
Level 2C 19 20 15 
Level 3 11 10 21 
Missing 6 3 2 


Key Stage 1 reading 


Level 1 and below 19 18 11 
Level 2A 21 22 25 
Level 2B 26 27 23 
Level 2C 14 15 11 
Level 3 14 14 26 
Missing 6 4 3 


Key Stage 1 writing 


Level 1 and below 24 22 14 
Level 2A 14 15 21 
Level 2B 27 28 29 
Level 2C 23 23 20 
Level 3 6 6 13 
Missing 6 5 3 
School n 94 532 16,255 
Pupil n 5,964 35,726 1,118,860 


Note: ‘All eligible pupils’ refer to all pupils in 532 schools that were eligible to be recruited into the trial. Trial 
participants includes both treatment and control group with maths SATS data. England provides figures for all 
state school pupils. 
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Table 23: Comparison of demographic characteristics of trial participants to (1) the population 
of eligible pupils and (2) all state school pupils in England 


Trial participants (%) All eligible pupils (%) | tate} E-Tare m9) 


Currenily eligible 

for FSM 

No 65 60 82 
Yes 35 40 18 
Gender 

Female 49 49 49 
Male 51 51 51 
EAL 

No 81 77 82 
Yes 19 23 18 
School n 95 532 16,255 
Pupil n 6,172 35,726 1,156,236 


Note: ‘All eligible pupils’ refer to all pupils in the 532 schools that were eligible to be recruited into the trial. Trial 
participants includes both treatment and control group and the N includes some participants for whom test data was 
missing, hence the slightly higher pupil number in comparison to Table 22. England provides figures for all state school 


pupils. 
Cost 


Table 24 shows that to buy the LRS intervention would cost a school £76.60 per pupil per year, over 
three years, based on one class receiving the intervention. This includes the handsets for all children, 
the software, the set up costs, and training for one classroom teacher. As the majority of the costs are 
per class, the cost per pupil over three years would remain around £70, even if the number of classes 
involved was larger (assuming 1 teacher per class). 


Additionally, the school would need to add the cost of staff cover for seven days for the first year for the 
classroom teacher to attend training and two days in the second year. The process evaluation suggests 
that in the first year most teachers considered the time it took to plan lessons using the LRS intervention 
to be greater than for normal non-LRS lesson planning. Part way through the intervention period, the 
developers supplied intervention class teachers with a tablet computer to assist mobility in the 
classroom when using the software, but the developers considered this an optional extra cost. 


In this evaluation, the costs for the programme were covered by the EEF grant. Schools only had to 
cover the costs of half their supply cover (the programme paid for the other half of these costs). These 
costs varied as schools had different arrangements for staff cover when teachers attended the training 
days. 
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Type of cost Cost Total cost We) f= evel) wm ol- 1 
over 3 years pupil per year 
over 3 years 

Handsets, software and | Start up cost, per class £1,600 £1,600 

hub 

Half-day technician time | Start-up cost, per school £80 £80 

to set up 

Training, 7 days in first | Start up cost per person £3,829 £3,829 

year trained 

Training, 2 days in One off cost per person £1094 £1094 

second year trained 

Access to Inside Running cost per school, £100 £300 

software (analytical optional, but optimum 

tool) 

Total £6903  £6903/3 years /30 
pupils per class= 
£76.70 per pupil 

per year 
(or £2301 per class 
per year) 
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Process evaluation 


Is the LRS intervention attractive to stakeholders? 
Attractiveness to teachers 


In general, the LRS was well received by the majority of teachers, with over 90% stating in the 2016 
teachers’ survey that it was a valuable addition to their teaching. Key aspects of the programme that 
teachers found positive are detailed below, taking into account data from across the process evaluation. 


Pupil engagement 


On the whole, teachers thought the LRS helped to sustain pupil engagement. High levels of pupil 
engagement were witnessed in class observations during the evaluation. Improved engagement was 
attributed by teachers to the following: 


e instant feedback which promotes independent learning; 


e the promotion of healthy competition on an individual pupil level (beating a previous score) 
and a class level (wanting to beat their friends); 


e the ability for pupils to control the pace of their learning and remain challenged; 

e encouragement of the less confident learners to participate by allowing anonymity and 
reducing copying; 

e the differentiation function which facilitated learning across the ability spectrum; 

e aculture of whole-class participation; and 

e adifferent, fun and fast, dimension in the classroom. 


Teacher assessment 


Teachers found it really valuable to be able to promptly assess understanding by the class, based on 
the rate and accuracy of pupil electronic responses. It allowed them to intervene at class, small group, 
or individual pupil level, in real-time and in a targeted fashion. This was considered important for 
maintaining momentum around an issue, and thereby facilitating learning. They compared this 
favourably to the traditional approach of taking books home to mark at the end of the day and the 
associated delay in feeding back to pupils and guiding future lessons. 


The LRS was also seen as an effective and sensitive way of rapidly and anonymously assessing pupil 
confidence and knowledge, before and after a topic, and also gathering pupil opinion and feelings. 


Many teachers welcomed the fact that copying was much less feasible with the handheld devices, as it 
is not possible for a child to see their neighbour’s screen, thereby ensuring that they were assessing 
each pupil’s own work. 


Automatic centralised data storage 


Being able to automatically export class results into a spreadsheet and use this to readily identify where 
more practice was needed, at a class and individual level, was considered very useful in terms of 
facilitating lesson planning and targeted additional input for individuals. It was also used as a record of 
pupil progress. The software used for this was Excel or INSIDE. 


Instant feedback to pupils 


In general, teachers were positive about the immediate, real time, personal feedback function. They 
thought this helped with pupil engagement, prevented misconceptions building up, and facilitated active, 
independent, and self-paced learning. With experience, many teachers were able to adapt the feedback 
to the handsets to be more nuanced, such as providing a series of hints and tips to facilitate independent 
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correction by pupils, as opposed to supplying the answer too readily. Drawbacks were, however, 
reported, particularly the sensitivity of the system to minor errors typed in by pupils. This resulted in 
false negative answers being fed back electronically to pupils leading to frustration and putting 
engagement at risk. 


Showing whole class activity on the class whiteboard 


The speed with which an aggregated public electronic display of work could be assembled and shown 
on the class whiteboard was viewed as valuable. For example, it allowed pupils to see that others 
shared their own misconceptions of a topic and enabled rapid collection or sharing of work such as a 
word bank. The manner in which LRS activity was publicly displayed varied. A key issue was whether 
work was shown anonymously or not. For example, teachers could choose to show names and rankings 
in a mental maths task or names during a word seeding exercise. Teachers were divided on the value 
of non-anonymity. A few were of the view that it created competition and that children were naturally 
competitive and motivated by this. In general, most teachers only used the public non-anonymized 
approach occasionally and selectively and otherwise made it an anonymous exercise to avoid 
demotivating and stigmatising lower achievers. 


Class behaviour management 


In general, due to the fact the LRS engaged pupils, it was perceived by teachers as a useful tool for 
managing behaviour. For example, teachers used it for quick-fire questions at the start of a lesson to 
settle a class down, or at the end of a lesson as a reward for behaving well. 


Self pacing and differentiation 


The fact that children could work at their own pace, progressing automatically through different levels 
of questions as appropriate, was very positively viewed by teachers. This allowed high attaining pupils 
to be engaged with automatic increasingly challenging tasks which released time for teachers to focus 
on the slower learners. These slower learners also benefitted from the differentiation feature by being 
able to work at an appropriate pace without the stigma of routinely finishing last, or holding the class 


up. 


The equipment 


The teachers saw the LRS as providing variety and fun; the allure of a gadget in a generation that is 
technology ‘savvy’. Class observations revealed how using a computer tablet enabled the teacher to 
move around the class interacting with pupils as they worked, rather than having to refer to the main 
desktop computer screen at the front of the class. 


Suitability of the system for different types of tasks 


The LRS was valued particularly for ‘quick fire’ revision, for example, mental maths and grammar. Doing 
this frequently using the handsets, to build speed and accuracy, was perceived by teachers to be more 
acceptable to their pupils than traditional methods. In general, teachers reported that they found the 
LRS more valuable for teaching maths than literacy; as explained below this was primarily due to the 
difficulties pupils experiencing using the devices for typing in lots of text. 


Suitability of the system for different types of pupils 


Many teachers thought the LRS was useful for both Year 5 and Year 6 pupils and that it had benefits 
across the ability spectrum. Teacher survey results suggested that it was considered slightly more 
suitable with high, rather than lower, achieving pupils. This was attributed to the fact that these pupils 
could be more effectively challenged and supported as a result of self-pacing and also teachers using 
the differentiation facility. 
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Training and support 


The Edge Hill training and support for the LRS was highly valued and well used. The introductory 
training was rated as ‘good’ or ‘very good’ by the majority of respondents (94%) as were the five half- 
termly training days (84%). Attendance at training session was high, despite the frequency with which 
these were run. The most common reason for teachers not attending training was competing demands 
on their time, such as preparing for Ofsted or SATS. 


Ad hoc training and support, via telephone, email contact, and site visits, were offered to schools 
individually. The project lead and head of training also carried out scheduled visits to all the schools. All 
survey respondents reported at least one visit to their school by the project team in the first year of the 
programme (range 1-5 visits) and 70% of respondents reported contacting the project team, on an ad 
hoc basis, by email or phone for support (range 1-15 contacts). Telephone and email support was 
highly rated by the vast majority of teachers (90% of respondents said ‘good’ or ‘excellent’) as was 
support via school visits (88% of respondents said ‘good’ or ‘excellent’). They found the project team to 
be easy to access whenever required, supportive, efficient, and very effective at resolving problems 
quickly. 


Resources 


The bank of resources (question sets) created by the Edge Hill team that teachers could draw on to use 
in their lessons was seen as critical to the feasibility of delivery. This was particularly key at the start 
and for those who were less confident using IT. The quality of resources was seen to improve over the 
programme through Edge Hill adaptations of those in the bank, teachers contributing their own 
resources for others to use, and one of the teachers on the programme creating a website that allowed 
teachers to make high quality resources relatively quickly. 


Teacher time and level of use 


Most teachers mastered the system during their first year of use. Once proficient, many teachers felt it 
saved time or led to more effective use of time, for example in terms of marking, recording, and meeting 
pupil needs across the ability spectrum. 


Support at whole-school level 


Most teachers appeared to feel that their involvement with the programme was well supported by the 
school senior leadership team (SLT) and by teaching colleagues. This is evidenced, for example, by 
the fact that reasons for missing training were rarely due to lack of permissions from the SLT and that 
some use of the handsets in other year groups took place over the two-year programme in nearly half 
the intervention schools. 


Attractiveness to pupils 


In general, Year 5 and Year 6 pupils liked the LRS. At the start of the focus groups, held in the first year 
of the programme, pupils were asked to say how they found the use of the handsets in class, using the 
options ‘really good’, ‘Ok’, or ‘not good’. Equal numbers of pupils said ‘really good’ or ‘OK’. One pupil 
did not think it was good. In Year 2 of the programme, more thought they were ‘OK’ than ‘really good’. 
No one said they thought they were ‘not good’. In the focus groups, while participants could find aspects 
of the handsets to critique (see section below), as a whole they were more positive than negative about 
their use. 


Engagement 


Using electronic equipment, as opposed to traditional pen and paper, was in general perceived by pupils 
as fun, ‘modern’, and inclusive. 
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Feedback 


Children liked the concept of receiving instant feedback on the handheld device. When well used by 
teachers—so that hints and tips rather than direct results were received—children found it supported 
their learning by alerting them quickly to where they were going wrong, and providing encouragement. 


Privacy and Competition 


Children were very positive about the privacy the LRS allowed. They reported that this gave them 
confidence in making contributions in class, in both academic and social matters, without the risk of 
embarrassment. 


Many children enjoyed the ability to compete with their own previous scores (for example in mental 
maths) and with their friendship group. Some were also relaxed about non-anonymous public displays 
of their results. In general, pupils were very mindful of the risk that the LRS could lead to unhealthy 
competition and were clear that good teacher management was critical to minimise this. 


Children’s positive comments about the LRS 


‘I like everything—it’s a lot easier and a lot quicker than having to put your hand up—often you don’t 
get picked [by the teacher] and you know you have the right answer and she says you can put your 
hand down now.’ 


‘| think the feedback is helpful, if it is just the teacher taking it home you have to wait a whole night 
for it to happen but it just tells you straight away so you can just correct what you have done wrong.’ 


‘Because it tells you if you have got the right answer which gives you more courage to do more 
questions like it says ‘well done’ or ‘excellent’ which makes you feel more happy about your answer.’ 


‘It feels quite private, like no one else can copy, so it feels quite secure ... so it’s all your own 
answers.’ 


‘If we are doing a test it comes up with our names but Miss covers our names. On the bar graph Miss 
can check who answered what—she doesn’t want to embarrass people.’ 


Aspects that were less attractive to teachers 


Approximately a quarter of teachers expressed a degree of ambivalence about the value added by the 
LRS compared with teaching resources they believed to be tried and tested, such as white boards or 
other recently-tested new interventions. This ambivalence was often linked to particular issues with the 
system, explained below, including concerns about evidencing pupil work, the sensitivity of the system 
and the impracticality of the handsets. In general, most teachers found ways of managing these 
drawbacks, for example by using the LRS selectively and in distinct ways. If these issues were resolved, 
many teachers, said they would consider the LRS a wholly positive asset. There were also teachers 
who acknowledged that their own limited engagement with, and use of, the intervention was a limiting 
factor. 


However, there was also some more fundamental disquiet among teachers. In the survey at the end of 
Year 1 of the programme, 25% of the 93 respondents agreed with the statement, ‘[The LRS] represents 
an approach to education that is potentially undesirable’. In the survey at the end of the programme, 
10% of respondents answered that they did not consider the intervention added value to their literacy 
teaching over and above other teaching methods. There was evidence that their view that the system 
did not capture processes, only outcomes, was a key concern for some of these least enthusiastic 
teachers. 
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Specific aspects that teachers found less attractive about the LRS are detailed below, taking into 
account data gathered from across the process evaluation. 


Evidencing pupil work 


In Year 1 of the programme it was a common teacher concern that working electronically using the LRS 
meant that it was difficult to readily demonstrate hard evidence of the work being done by pupils. This 
was a particular concern in the context of Ofsted inspections. Most teachers felt they had addressed 
this, by the end of Year 1, by moving towards routinely using the LRS in combination with writing in 
exercise books. Pupils used paper and pencil for working out their answers and then submitted them 
electronically via the LRS. This was generally viewed by teachers as an acceptable solution. 


False negative feedback on handsets 


The sensitivity of the system created difficulties. If a child misspelt a word, made inaccurate use of 
punctuation, or misused upper- or lower-case letters, the feedback received was that they were wrong 
even if essentially the answer was correct in its lesson content (and would have been marked as right 
by most teachers). Some schools used this as a way of encouraging precision, for example with 
punctuation, but most believed it was demoralising and frustrating for pupils and at worst led to 
disengagement. 


Resources 


Criticism of the resources in the resource bank included the multiple errors within them which created 
confusion in the classroom, that the level resources were pitched at was often higher than the ability of 
many classes, and that predetermined feedback did not necessarily match why a child got an answer 
wrong. 


Creating their own good quality LRS resources, with specific feedback, was found to be very time 
consuming by teachers, particularly at the start of the programme. Teachers commented that if 
insufficient time and care were taken creating these, there was a risk that inaccuracies crept in which 
resulted in frustration and rapid disengagement by pupils. With particular types of resources, such as 
those for guided reading, time spent preparing versus benefit gained was questioned by some staff. 


The equipment 


The main criticisms of the equipment were: 


e the style of the handheld device and size of the key pad; and 
e problems with the technology. 


In terms of size, the key pad and buttons were small. This led to problems with accuracy of typing and 
accessing punctuation, particularly for children with fine motor skill problems. Teachers said the hand- 
held devices were relatively old fashioned and basic compared to the personal electronic devices (Such 
as mobile phones) that many children currently have access to. They thought this might lead to 
diminishing engagement by children over the longer term. 


In terms of the technology, initial local problems were generally resolved with the support of the Edge 
Hill team. Teachers mentioned some ongoing frustrations which wasted time, such as the regularly 
required registering of devices, dealing with ‘dead’ batteries, and occasional system crashes. Also, if 
the system was not functioning effectively (for example due to IT problems or resource errors) children 
lost interest and patience and teachers described having to abandon their planned use in the middle of 
a lesson and resort to other approaches. 


Some schools had specific IT issues. For example, teachers in one school reported that the tablet would 
not work due to Wi-Fi issues. Another school had very limited use in the first year due to a combination 
of local IT problems and staff turnover. 
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Class behaviour 


When the LRS was used in the first year, the tendency for pupils to compete created difficulties, with 
pupils racing to see who could finish a task first, regardless of accuracy, and then being disruptive as 
they celebrated their ‘success’. This was particularly the case with timed exercises, such as mental 
maths and SPAG. Once this problem was addressed, by teachers stressing the need for accuracy, it 
became less of a concern. Many teachers stopped using the ranking facility to make it less easy for 
pupils to compare notes as to where they had come in an exercise. 


Lack of suitability for different types of ‘tasks’ and pupils 


The issues around accuracy of typing led to the LRS being generally considered as more useful for 
maths or English questions that were not reliant on much inputting of text, such as SPAG and 
‘wordseeding’. The small screen also meant that they were considered less suitable for work requiring 
pictures (including English SATs questions). Some teachers acknowledged that they predominantly 
only used in it one subject, usually maths. In addition, teachers with high levels of EAL children in their 
class commented that the LRS was not appropriate for English lessons because these children were 
not able to participate effectively. 


It also emerged that some teachers did not consider the LRS promoted class discussion as effectively 
as other teaching strategies and some questioned the detrimental impact of this on higher achievers 
who particularly require opportunities for exploration and debate. 


In interviews, teachers also often mentioned an occasional individual pupil who was less enthusiastic 
about the tool than most and there were rare examples given of where this applied to specific whole 
classes (for reasons teachers could not identify). 


Teacher time and level of use 


A few teachers said they were not able to dedicate enough time to learning and practising with the LRS 
in order to become proficient. As a consequence, they were not able to progress from using the system 
in a relatively basic way. Most who fell into this category regarded themselves as under-confident with 
technology. These teachers, in general, were keen to progress but were sceptical as to whether they 
would be able to do so due to time constraints. Level of use was linked by many to potential benefits. 


Results rather than processes 


The fact that the electronic data was provided to teachers as results rather than processes was a major 
disadvantage for a few teachers. They said this limited its value for assessing the exact nature of the 
support required and made it an unacceptable intervention, providing no additional value to other 
methods such as individual pupil whiteboards. Others considered this a significant flaw, but thought— 
particularly if used at an advanced level and in conjunction with recording on paper as described 
above—the LRS remained of value. 
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Teacher comments about the LRS 


[The LRS] enables every child to be involved and work at their own pace without shame or 
embarrassment’ (teacher survey). 


I’ve taught some of these children for three years now and [since using the LRS] | know them much 
better—what their strengths and weaknesses are, etc.’ (teacher interview). 


‘There is nothing that the [LRS] does that you can’t do with whiteboard and pen—with the interactive 
games and software that is out there the children can just do it in school with a whiteboard and pen. 
However, the actual getting it out and doing work on it is probably a better way to engage them—the 
actual learning side of it could be done on a white board and a pen’ (teacher interview). 


‘So rather than saying that it will revolutionise teaching ....... careful about making fantastical claims 
about it because it is how it is used that is important’ (teacher interview). 


‘Yes (it will have impact) if well used, but I’m not sure | use them very well’ (teacher interview). 


‘To have a really well differentiated quiz that meets exactly what you want your class to do that day 
takes a really long time [to develop]’ (teacher interview). 


‘lwas told by Edge Hill that it was a teaching tool when what | was seeing was them as an assessment 
tool—then | saw that colleagues who were more confident were embedding videos, using ad hoc 
questions etc. to bring lessons alive. | didn’t have the skills to do this but | saw the potential’ (teacher 
interview). 


‘| want to understand their conceptual development around a subject, not whether they have got the 
answer right’ (teacher interview). 


Aspects that were less attractive to pupils 


As stated above, only one of the focus group participants did not like the intervention, but many pupils 
expressed some reservations about it 


‘I do enjoy [LRS] but | couldn't live with them every day, | do get frustrated with them. | must 
admit writing is quite nice’ (Year 6 pupil). 


The equipment and resources 


The perceived limitations of the equipment was the strongest negative theme from the children’s focus 
groups. These included battery failures, IT issues, and the small size of the keyboard, keys and screen. 
As with the teachers, the keyboard and keys size emerged particularly strongly as a negative pupil 
theme due to the fact they often got feedback that an answer was wrong due to a typing error rather 
than a lack of knowledge. Similarly, they discussed their frustrations when there were errors in the 
resources which similarly resulted in ‘false negative’ outcomes. Children found these issues, which they 
perceived as ‘not their fault’, very frustrating and demoralising. Many of those who answered that the 
LRS was ‘OK’ said their acceptability rating would be raised if these functional factors could be 
improved. 


They were also frustrated by the fact that, unlike a teacher, the LRS could not discern between an 
answer that was good enough to be scored as correct in the SATs, despite not being entirely correct, 
and an answer that was not good enough. 


Feedback 


Even though pupils liked the feedback function, it was a common theme that they disliked answers 
being provided too readily. They wanted feedback in the form of a series of tips and prompts for solving 
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the problem independently, with the answer only provided when they had had to persevere 
independently. It appeared that the teachers who were least proficient on the system were more inclined 
to provide electronic results more readily than pupils wanted them. 


Competition and privacy 


While children liked competing against themselves (for example, their previous spelling test scores) 
and occasionally with their peers, in general they disliked the scale of competitive behaviour the LRS 
had the capacity to engender. 


Children discussed at length the practice, used by some teachers, of displaying whole-class results 
non-anonymously on the whiteboard. Many children really disliked this approach, an opinion expressed 
both by those who were accustomed to performing relatively poorly, and those who were used to doing 
well but were concerned for their less able friends. They saw it as poor practice by teachers with the 
potential to embarrass pupils and even put them at risk of bullying. 


Pupil comments 


‘Don't like it when it tells you you’re wrong and you aren’t—this happens about once a week (Year 6 
pupil). 


‘I don't really like that [displaying of whole class results], | like it so that only me and my teacher knows 
what my progress is and what I have done’ (Year 5 pupil). 


‘On the [LRS] if you get a question wrong it just tells you ‘incorrect’ and then it will just tell you ‘the 
correct answer is’ and tell you this without telling you how to work it out, we are not learning anything 
because it is telling you the answer. So, they should tell you how to work it ouf (Year 6 pupil). 


‘| think you spend more time with a teacher if you’re using a book rather than [LRS] because when 
she comes round to have a look at your work—if you have done something wrong (looking in your 
book) she will say something like ‘check number 13 again, | think you’ve done the calculation wrong’ 
then you look at number 13 and you see what you’ve done wrong and it helps you, whereas with the 
[LRS] she looks on the screen but she can’t talk to you because everyone else is working (Year 6 


pupil). 


Barriers to delivery 


The equipment 


The physical quality of the handheld device was criticised by teachers and pupils alike. The small 
keyboard and small screen size were found to be particularly unattractive, limiting its suitability for a 
range of tasks. 


Technology 


Frustrations with the technology were generally minor, though not insignificant, for example the 
batteries in the hand-held devices would ’die’ without warning and the devices needed registering at 
the start of sessions. Occasionally technological hitches were major and severely impacted delivery for 
a period of time. The inflexibility of a machine, compared with a skilled person, was a barrier for pupils 
(in terms of false negative feedback) and teachers (in terms of lack of data on pupil processes rather 
than outcomes). 
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Teacher time and skill 


For some teachers, learning and delivering the LRS required time and IT aptitude that they did not feel 
they had the capacity for. This reduced the quantity and quality of delivery. The requirement to submit 
three resources per week to Edge Hill to show they were complying with the expected use of the 
programme created added pressure for many staff. 


Fidelity 


Uptake and retention of schools in the intervention 


Out of the 49 schools randomised to the intervention, one did not ultimately take up the offer of the 
intervention training or handsets. They initially wanted to delay until the second year of the intervention 
because of IT issues, but chose not to join in the second year due to ongoing staffing problems. A 
second school took up the offer of training but decided to drop out of the intervention delivery after the 
first term in the first year, citing difficulties with staffing. 


The remaining 47 intervention schools all attended training and delivered the intervention in the first 
year. In the second year of the intervention, a further two schools stopped delivering—both because of 
staffing issues. 


Training—uptake 


The first aspect of the intervention was training teachers to use the system. As described in the 
intervention section earlier in the report, the schools were offered two days of baseline training and five 
half-termly training days in the first year of the intervention. In the second year, teachers who had 
previously attended training had one top-up training day per term, and new teachers had five days of 
training. Attendance data shows that the first year of training was well attended, with a global average 
of schools taking up 89% of the training on offer for the 47 schools that delivered the intervention. For 
21 schools, all relevant teaching staff attended 100% of the training, for a further 20 schools across 
their staff 75%-93% of available training was attended; and for the remaining six they ranged from 50% 
- 70% attendance. In the second year of the intervention, there was a slight reduction in attendance of 
training sessions — with 84% uptake across schools, and with 10 having 100% teacher attendance at 
the relevant number of sessions. Edge Hill offered to come to provide additional support to teachers 
who missed sessions. 


The most common reason for teachers not attending training was competing demands on time, such 
as preparing for Ofsted or SATs; some of the teachers started the programme late and were not 
therefore available for the early training. The introductory training was rated as ‘good’ or ‘very good’ by 
the majority of respondents (94%) as was the half termly training (84%). 


Usage and skill using the LRS 


The process evaluation gathered intervention usage data from two main sources: (1) Edge Hill weekly 
monitoring data, where resources used were uploaded, and (2) information from teacher surveys. 


Monitoring data about intervention delivery 


In some schools, teachers reported that they found it difficult to find the time to upload their resources 
to Edge Hill, despite having been using the handsets. The Edge Hill team vigorously pursued the 
monitoring, but acknowledged that there would be some limited degree of under-reporting of usage in 
this monitoring data, so this should be noted when viewing this data. 


The intervention for cohort A (1 year of intervention) was intended to involve the use of the LRS 
handsets in lessons for 25 weeks of the academic year (October to early May, culminating prior to SATS 
testing mid-May 2015). In practice, the number of weeks of intervention was reduced in most schools 
in this cohort, for a variety of reasons. Some schools had an additional week of half term, bringing the 
number of weeks possible for using the handsets to 24. Additionally, the majority of teachers of this 
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cohort reported reasons for non-delivery on weekly monitoring collected by Edge Hill (illness, 
assessment weeks, residential trips, Ofsted inspections and jury service all limited the number of weeks 
when delivery occurred). Additionally, technological problems hampered delivery in two schools, 
delaying the start by several weeks. These reported issues curtailed delivery, with less than half the 
schools delivering more than 20 weeks and the mean number of weeks of delivery being 19 (see Table 
25). 


For cohort B (receiving 2 years of intervention), the period of time for handset use was for 32 weeks in 
the first academic year (October to July) and 26 weeks for the second (September to mid-May). 
Monitoring data suggests that delivery tapered off in the second year of the intervention, especially in 
the lead-up to SATs. Over a quarter of schools (28%) reported delivering the intervention for 9 weeks 
or fewer in the second year. The average number of weeks where there is monitoring evidence of 
delivery was 20 weeks in Year 1 and 15 in Year 2 (see Table 25). Staffing issues remained the critical 
issue regarding low use — i.e. staff illness, change in role of trained teacher. 


Table 25: Number of weeks when any delivery was reported within cohort—monitoring data 


Number of weeks LRS_ Cohort A (1 year of Cohort B (2 years of intervention) 
ULY=W c=) ole) a (-to ll oht] intervention) 
Yes aTofe) | 

Year 6 2015 Year 5 2015 Year 6 2016 
0-9 weeks 0 schools 4 schools (9%) 13 schools (28%) 
10-19 weeks 27 schools (57%) 17 schools (36%) 17 schools (36%) 
20-32 weeks 20 schools (43%) 26 schools (55%) 17 schools (36%) 
Expected number of 25 weeks 32 weeks 26 weeks 
weeks of delivery 
Mean weeks delivered | 18.75 weeks 20.3 weeks 15.25 weeks 


Monitoring data also provided information about the number of resources uploaded and the subjects of 
lessons that were covered. This information was used to develop a ‘dose’ score for schools, by cohort 
and by subject (maths, English) that was utilised in the impact analysis. 


Survey data 


Teacher survey data provides additional information about delivery of the intervention, including the 
amount of time spent using the LRS, the types of lessons, and the degree of proficiency on them. Those 
who responded to the survey, especially in the second year, were more likely to be those who also 
returned monitoring resources. The survey data may therefore provide a somewhat skewed positive 
picture of usage. 


Using the LRS when teaching Maths 


Among the teachers who completed the surveys, the average time spent weekly teaching maths using 
the LRS was greater for Year 6 teachers than for Year 5, with the most time-intensive use in the second 
year of cohort B (Table 26). 
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Table 26: Reported minutes of teaching using LRS for maths in an average week (2016)— 
teachers’ surveys (expected at least 20-30 minutes per week) 


ele as ejgte Ca ejate > Pd e E = e 


Year 6 2015 Year 5 2015 Year 6 2016 
(n = 54; (n = 39; (n = 45; 
missing = 0) missing = 3)* missing = 1) 
Never 0 (0%) 0 (0%) 0 (0%) 
1-29 minutes 16 (30%) 18 (46%) 10 (22%) 
30-59 minutes 19 (35%) 12 (31%) 9 (20%) 
1-3 hours 19 (35%) 8 (21%) 19 (42%) 
> 3 hours per week 0 (0%) 1 (2%) 7 (16%) 


* Includes teachers who taught combined Year 5/6 classes in 2015. 


In the teaching of maths, by the end of the second year for Cohort B, of the teachers who responded to 
the survey, 2 (4%) said that they were proficient on the LRS at a basic level; 27 (60%) said they had 
intermediate proficiency, and 16 (36%) said they were advanced users (who were able to do all the 
things taught at the Edge Hill training as well as additional self-taught ways). 


Table 27: Ways LRS was used by teachers—maths 


Type of use—maths me) 


Beginning of the day, before lessons begin (individual) 34 (74%) 
Warm-up at start of a lesson (whole class) 39 (85%) 
Main focus of lesson (whole class) 27 (59%) 
Used intermittently through lesson (whole class) 15 (83%) 
End of lesson assessment of understanding 29 (63%) 
Tool for working with small groups 15 (33%) 
Working in pairs or individuals with teacher 7 (15%) 


Teachers used the LRS across the range of maths domains. The components which teachers reported 
using very often (or quite often) included mental maths and number (Table 28). These teachers said 
they found the handsets most useful when teaching times tables, mental maths, fractions, and measure. 
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Table 28: Components of maths where LRS was used often by teachers 


WN ENdatemexeynny fey al-val cy Often used LRS* 
n(%) 

Mental Maths 45 (100%) 

Number 44 (98%) 

Graph and Data 30 (67%) 

Measure 30 (67%) 

Statistics 18 (40%) 

Algebra 15 (84%) 

Geometry 8 (18%) 


* Very often’ and ‘quite often’. 


Using LRS when teaching English 


As with maths teaching, the time reported by teachers who completed the surveys, in terms of average 
weekly teaching of English using the LRS, was greater for Year 6 teachers than Year 5 with the most 
time-intensive use in the second year of cohort B (Table 29). 


Table 29: Reported minutes of teaching using LRS for English in an average week (2016)— 
teachers’ surveys (expected at least 20-30 minutes per week) 


( Year 6 2015) Year 5 2015 Year 6 2016 
(N= 51; (N = 36; N = 45; 
3 missing) 6 missing) 1 missing) 
Never 0 (0%) 0 (0%) 2 (4%) 
1-30 minutes 21 (41%) 19 (53%) 13 (29%) 
31-59 minutes 18 (35%) 11 (26%) 12 (27%) 
1-3 hours 12 (24%) 6 (14%) 13 (29%) 
> 3 hours per week 0 (0%) 0 (0%) 5 (11%) 


*includes teachers who taught combined Year 5/6 classes in 2015. 


In the teaching of English, by the end of the second cohort year, of the teachers who responded to the 
survey, 4 (9%) said that they were proficient using the LRS only at a basic level, 30 (68%) said they 
had intermediate proficiency, and 10 (23%) said they were advanced users (who were able to do all the 
things taught at the Edge Hill training as well as use additional self-taught strategies). The self-reported 
proficiency level with the LRS was thus lower in English than in maths. 
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Table 30: Ways LRS was used by teachers—English 


Type of use—English n (%) 
Beginning of the day, before lessons begin (individual) 24 (52%) 
Warm-up at start of a lesson (whole class) 32 (70%) 
Main focus of lesson (whole class) 22 (48%) 
Used intermittently through lesson (whole class) 22 (48%) 
End of lesson assessment of understanding 21 (46%) 
Tool for working with small groups 14 (30%) 
Working in pairs or individuals with teacher 6 (13%) 


Teachers reported using the LRS across the range of English domains. The components for which they 
reported using the LRS very often or quite often included grammar and word classes (Table 31). These 
teachers said they found the handsets most useful when teaching grammar, spelling, punctuation, and 
word classes. 


Table 31: Components of English teaching where LRS was used often by teachers 


English components Often used LRS* 


n (%) 

Grammar 42 (96%) 
Word classes 41 (93%) 
Punctuation 40 (91%) 
Spelling 35 (80%) 
Sentence level 31 (71%) 
Writing 16 (81%) 
Reading comprehension | 6 (18%) 


* Very often’ and ‘quite often’. 
Formative findings 


Most teachers and children were very positive about the LRS and did not have suggested changes. 
The primary way it was suggested that the LRS programme could be improved, by those in the minority 
who put forward such ideas, was to improve the handsets by making the keypads and screens such 
that they reduce the risk of typing errors and are more attractive to pupils in relation to other technology 
they access. 


The process evaluation findings suggest that pupils might be happier with the handsets if they were 
used by teachers to give feedback to pupils individually, or anonymously, rather than in a whole-class 
named ‘league table’. Although some teachers thought the children were spurred on by competition, 
this view was not shared by pupils. As such, this modification of usage could be considered. 
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Additionally, some teachers would appreciate if training gave space for pedagogical discussion about 
the ways that using the handsets changes the nature of their teaching. Addressing such issues head 
on, and taking these seriously, could have led to more engaged or creative use of the LRS by some 
teachers. 


Finally, process evaluation findings suggest that the training could also provide some guidance to 
teachers regarding the importance of avoiding errors in the pre-programmed answers within lessons on 
the LRS, as these errors can reduce pupil engagement with the system. 


Control group activity 


Headteachers from 44 control schools were sent a very short email survey in June 2016 which included 
three questions about their activities in their Year 6 classes during this academic year (the second year 
of the intervention). After sending three reminders, including sending to alternate contacts from control 
schools, we received information from 24 control schools, a response rate of 55%. 


We asked whether their Year 6 class had used any handheld electronic feedback devices during this 
academic year. While 22 out of 24 who responded said that their Year 6 classes had not used such 
devices, two schools indicated that they had used them. One said that this was ‘briefly, wnen handsets 
were loaned to the school’. The second explained that these were not the same type of handsets and 
were used on an ad hoc basis. While this suggests that there was some minor usage of electronic 
feedback devices in a small number of schools, they were not exposed to the multiple training days, 
the curriculum resources, or the particular technology provided by the Edge Hill team. We cannot be 
certain whether any of the non-responding control schools used such devices, but we do know that 
none of them received the Edge Hill package. 


Control schools used a variety of additional inputs to improve maths attainment with their Year 6 
classes. In the current academic year, of those schools where a response was provided: 


e 19 (79%) were providing booster revision sessions within lesson time; 


e 19 
e 20 
e 11 
e 4(17%) had an additional maths specialist working with the children. 


) 
79%) were using booster sessions outside of lesson time; 
83%) were providing additional teacher or TA support in the classroom; 


amy 


44%) were using online maths packages (such as Mathletics); and 


One school had begun the Maths Inspire scheme across their school. Only one of the 24 responding 
control schools said they had used no additional inputs to raise maths attainment. This compared with 
the LRS intervention group where a lower proportion of those schools (n = 40) that responded had 
implemented these additional types of additional inputs to raise maths attainment: 

e 16 (40%)—booster sessions in lessons; 

e 23 (58%)—booster sessions outside lessons; 

e 16 (40%) additional teacher/TA support in lessons; 

e 8 (20%)—online packages; and 

e 5(13%)—maths specialists. 


Six of the responding intervention schools said they used no additional inputs (beyond the LRS) to 
improve maths. 


In terms of improving literacy attainment, additional inputs were also reported to have been used by 
control schools who responded to the survey: 


e 20 (83%) of those who responded were using booster revision sessions within lesson time; 
e 19 (79%) were using additional booster sessions outside of lesson time; 
e 4(19%) were providing additional teacher or TA support in the classroom around literacy; 
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e 9 (88%) were using online literacy software; and 

e 3(13%) were utilising an additional literacy specialist. 
Only one school reported that it had used no additional inputs to raise literacy attainment. As with maths, 
in most areas, lower proportions of the LRS intervention schools who responded to the survey said they 
had used these types of literacy inputs: 

e 13 (33%)—booster sessions in lessons; 

e 18 (45%)—booster sessions outside of lessons; 

e 3(8%)—online literacy software; and 

e none were using literacy experts. 


Only additional teacher TA support was higher than in control group: 14 (85%). Overall, 9 (23%) said 
they were using no additional inputs (beyond the LRS) to raise literacy. 


The response rate from control schools was incomplete, and it is not possible to draw definitive 
conclusions as a result. From those who responded, it appears that control schools were undertaking 
more additional inputs with their Year 6 classes to raise maths and literacy attainment than the LRS 
intervention schools. If the control schools were doing more additional interventions, on average, than 
the intervention schools, this could have the effect of diluting the observed impact of the LRS. 
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Conclusion 


Key conclusions 


The project found no evidence that the LRS improved Key Stage 2 results in maths and 
reading for either cohort. 


The project found no evidence that the LRS improved the average scores of boys, girls or 
pupils who have ever been eligible for Free School Meals. 


Classroom teachers and children were generally positive about the LRS. Teachers 
welcomed the ability to quickly assess pupil responses for certain types of tasks and give 
instant feedback. They felt that the LRS helped to engage pupils and allowed different 
pupils to work at their own pace. 


Some teachers and pupils felt that the inability of the system to let children ‘show their 
working’ was a weakness. There was also a concern that incorrectly programmed or over- 
specified answers meant that pupils occasionally received negative feedback when in fact 
their answer was wholly or partially correct. There was some criticism from pupils and 
teachers that the small size of the handset made it difficult to type properly. 


The intervention was variably implemented, with a number of teachers not meeting the 
weekly targets for usage. Reasons given for this included staffing issues, lack of time, and 
the inability to use the system to its full potential. However, even in schools with high usage, 
the analysis did not find evidence of an impact. 


Interpretation 


This study hypothesised that the use of the LRS by Year 5 and Year 6 classroom teachers for one or 
two academic years would increase children’s achievement in maths and English. This RCT found no 
evidence of such an effect. Additionally, no effect was found for children who were ever eligible for FSM 
or for boys or girls. 


These results run counter to the previous research conducted by the IEE at York University (Sheard et 
al., 2011 and 2013). This previous research looked at very focused use of these same sort of electronic 
handsets, but for a more limited time (12 weeks) and on only one specific topic at a time. This Edge Hill 
LRS intervention was a more diffuse one, asking teachers to use the handsets with both maths and 
English teaching. Additionally, as detailed below, the process evaluation showed that use of the 
handsets over the entire school year (or two years, depending on the cohort) meant that the 
engagement with the intervention needed to be maintained for a considerably longer time by teachers. 
This means it could have been affected by usage by teachers tapering off over time, as well as the 
inevitable conflicts within a primary school year (in terms of staff illness, turnover, assessment practice, 
and other school priorities). Additionally, this evaluation had delayed measurement compared to other 
research that conducted the testing immediately after the end of the intervention period with a targeted 
test specific to the lesson skills for which the handsets were being utilised. In contrast, in this evaluation, 
high stakes assessments were used which had a broader focus, and were conducted at a time when 
use of the LRS was declining in the lead up to the exams. 


The process evaluation findings suggest that the programme was well liked, but that teachers used the 
LRS with variable intensity, and generally less than expected by the developers. Monitoring data 
suggests that the frequency of use dropped off in most schools near the end of each intervention year; 
and while there were teachers who self-reported greater intensity of use in the second year, the 
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monitoring data shows that the overall frequency of use was generally lower in the second year of the 
intervention, with a greater number of schools not delivering at all, or for considerably fewer weeks. A 
longer intervention such as this one exposed the programme to many changes in staffing and senior 
management within schools — priorities shift; teachers become ill, or other activities intervene over the 
course of time. Counter to this, some teachers embedded their use in the second year and used the 
handsets more intensively and with greater confidence. However, the process evaluation found that 
frequency of use dropped off in some schools because aspects of the intervention became less 
attractive to pupils (notably the size of handset). Additionally, over time many teachers restricted their 
use to specific tasks where they deemed it most useful and did not use it more broadly. 


The process evaluation showed that often this meant that teachers used the LRS for self-paced 
repetitive revision-type tasks, rather than for whole-class lessons. One such frequent use of the LRS 
was in maths for quick-fire times tables practice. However, rapid recall of times tables does not currently 
get directly measured in KS2 SATs, although the U.K. government has announced that such a times 
tables test will be brought in beginning with SATs 2019. Thus, although much use of the handsets was 
undertaken in this sphere, we could not determine whether the intervention had an impact on times 
tables performance. However, similar self-paced revision use with grammar, spelling, and punctuation 
was not shown to lead to different results between the two arms of the trial. 


Limitations 

This study had some measurement attrition (9% of pupils were not included in the analysis), but was 
generally robust and had enough power to show difference. 

Schools within this sample had a relatively low percentage of EAL students, so the results may be less 
generalizable for schools with a highly diverse language population. 


Future research 


Future research could look at changes to the handset size and user acceptability; and, based on the 
learning from this evaluation, designing and testing alternative training and support for ensuring fidelity 
and increasing use of the LRS. Additionally, given the imminent changes to the U.K. government testing 
at KS2 around multiplication tables, future research should be conducted that answers the question: 
Can the LRS impact on attainment results on times tables? 
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Appendix A: Memorandum of Understanding 


‘Learner Response System Technology Research Programme’ (LRS) 
Memorandum of Understanding/National Pupil Database (NPD) agreement form 


This Memorandum of Understanding (MoU) establishes a partnership between: 
Edge Hill University 
Institute of Education 
And 
{Name} Primary School 


for the delivery of the Education Endowment Foundation funded Learner Response System 
Project, May 2014 to July 2016 


Purpose and Scope 


Edge Hill University, Institute of Education and {name} Primary School agree to work 
together to deliver the Learner Response System research project on behalf of the 
Education Endowment Foundation. 


Responsibilities 


Each party will appoint a person to serve as the official contact and co-ordinate the activities 
of each organisation in carrying out this MoU. The initial appointees of each organization 
are: 


Edge Hill University 


Jan Jackson 

Project Leader, EEF LRS Project 

Edge Hill University, Faculty of Education 
jacksonj@edgehill.ac.uk 


Institute of Education 


Meg Wiggins 

Director of Evaluation 

Social Science Research Unit, Institute of Education 
m.wiggins@ioe.ac.uk 


Name of School 
If you have any queries about the evaluation, please contact Meg Wiggins at the loE at 


m.wiggins@ioe.ac.uk. Any queries relating to the LRS programme can be directed to 
Jan Jackson at jacksonj@edgehill.ac.uk 


Edge Hill A Education 


Endowment 


University Foundation 


Named contact 
(named) Primary School 
The organisations agree to the following tasks for this MoU: 


Edge Hill University will: 


e Recruit the requisite number of schools to make the project viable. 

e Liaise with IOE to provide names of schools eligible to take part in the randomisation 
exercise. 

e Ensure key data (UPN numbers) for all schools is collected, collated and sent 
securely to the Director of Evaluation at the Institute of Education (IOE). 

e Ensure all schools sign this Memorandum of Understanding. 

e Advise schools in a timely manner whether they are part of the intervention or 
comparison groups. 

e Agree to abide by the conditions set out in this MoU. 


In addition, for schools who are part of the intervention group: 


e Organise and deliver a conference day for lead contacts in schools to provide further 
information about the LRS and how the project will run. 

e Provide each school with the necessary hardware and software to use LRG in all 
Year 5 and 6 classes. 

e Identify small, locally-based cluster groups of schools for delivery of training. 

e Liaise with schools to source and fund venues for training. 

e Provide two days of baseline training in May/June/July 2014 for all year 5 and 6 
teachers and pay schools supply cover for this. 

e Provide the school with resources to deliver in school before the end of the 2013-14 
academic year in order to trial use of the LRS. 

e From September 2014, provide one day of training per half term for all teachers, and 
pay supply cover for half of this. 

e Produce a bank of resources for teachers to use with the LRS, identifying the tasks 
which must be carried out each half term as a minimum requirement of the project. 

e Regularly visit schools to advise and monitor use of the LRS. 

e Provide technical and professional support to allow teachers to become proficient in 
the use of the LRS in their classrooms. 

e Setup and monitor a website to facilitate support and also networking between 
schools. 


In addition, for schools who are part of the comparison group: 


e Regularly update schools on the progress of the research project. 


If you have any queries about the evaluation, please contact Meg Wiggins at the loE at 
m.wiggins@ioe.ac.uk. Any queries relating to the LRS programme can be directed to 
Jan Jackson at jacksonj@edgehill.ac.uk 


Edge Hill A Education 


Endowment 


University Foundation 


Institute of Education will: 


Take receipt of school names and key data from Edge Hill University (EHU). 

Carry out the randomisation exercise and advise EHU of the outcome in a timely 

manner. 

Undertake evaluation by: 
The independent evaluation by the Institute of Education requires the 
information detailed in order to conduct a statistically robust evaluation of the 
LRS programme. Questionnaire and other test data will be matched with the 
National Pupil Database and shared with the IOE and the Education 
Endowment Foundation for research purposes. Pupils’ test scores and any 
other pupil data will be treated with the strictest confidence. No individual 
school or pupil will be identified in any report arising from the research. 

Observe selected lessons, carry out focus groups with pupils and interview key 

teachers. 

Construct a logic model to evaluate the effectiveness of the project. 

Provide all schools with a certificate of research participation. 

Agree to abide by the conditions set out in this MoU. 


{named) Primary will 


Provide identified key information on pupils within Florence Melly Primary (UPN 
numbers) 

Agree to take part in a randomisation exercise to determine whether name of school 
will be part of the intervention group or the comparison group for the purposes of this 
project. 

Provide consent for the evaluation team at the Institute of Education to access pupils’ 
school records held on the National Pupil Database (NPD). 

Agree to abide by the conditions set out in this MoU. 


In addition, schools forming part of the intervention group will: 


Attend the Conference Day in May 2014. 

Ensure all Year 5 and 6 teachers attend the two days baseline line training in 
May/June/July 2014. 

Ensure all Year 5 and 6 teachers attend the half termly training days, arranging 
supply cover as appropriate and funding half of this. 

Ensure Year 5 and 6 teachers carry out the tasks provided at the baseline training to 
trial use of the LRS in school. 

Take receipt of all the hardware and software and ensure that this is installed in the 
appropriate classrooms and that handsets are registered to individual learners. 
Liaise with EHU to assist in the sourcing of venues for training. 

Undertake to deliver the minimum number of tasks using the LRS each half term. 
Provide teachers with the time and support to enable them to undertake their role in 
the project and embed use of the LRG into their classroom practice. 


If you have any queries about the evaluation, please contact Meg Wiggins at the loE at 
m.wiggins@ioe.ac.uk. Any queries relating to the LRS programme can be directed to 


Jan Jackson at jacksonj@edgehill.ac.uk 
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Endowment 


University Foundation 


e Ensure teachers undertake evaluation as and when required by EHU (normally at the 
end of each training day). 
e Allow access to IOE researchers to observe some lessons, carry out focus groups 
with pupils and interview key teachers 
e Allow access to EHU staff to observe lessons and provide support and guidance for 
teachers. 
In addition, for schools who are part of the comparison group: 


e InJune 2015, allow Year 5 and 6 teachers to complete a short on-line survey 
about their previous experience of using electronic feedback tools and other 
approaches to feedback in the classroom. 


Signatories 


Signed on behalf of Edge Hill University 


Jan Jackson, Project Leader EEF LRS Project 


Signed on behalf of Institute of Education 


Meg Wiggins, Director of Evaluation 


Signed on behalf of {named} Primary School: 


| understand and agree that (please tick each box): 
| | The school can withdraw from the research at any time 


The school consents to the use of its year 5 and 6 pupils’ National Pupil 
[ Database data for the purposes of this evaluation. 


If you have any queries about the evaluation, please contact Meg Wiggins at the loE at 
m.wiggins@ioe.ac.uk. Any queries relating to the LRS programme can be directed to 
Jan Jackson at jacksonj@edgehill.ac.uk 


Endowment 


University Foundation 


Edge Hill A Education 


[| Data collected as part of the evaluation can be matched to individual NPD 
records, and that this data can be shared with the IOE and EEF for research 
purposes (at a level of ‘Tier 1’ access). 


[J The school will distribute IOE information and consent letters to parents when 
required. 


[| The school will complete a School Data Form of key information on year 5 and 
6 pupils and send it (securely, electronically) to jacksonj@edgehill.ac.uk by 11 
April 2014. 


a teacher Date: 
signature: 


Please sign and return this form to Jan Jackson, Faculty of Education, Edge Hill 
University, St Helens Road, Ormskirk L39 1LF or jacksonj@edgehill.ac.uk by 11 April 
2014 


If you have any queries about the evaluation, please contact Meg Wiggins at the loE at 
m.wiggins@ioe.ac.uk. Any queries relating to the LRS programme can be directed to 
Jan Jackson at jacksonj@edgehill.ac.uk 
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Appendix B: Parental opt out consent form 


Dear parent/carer 

This year your child’s school has been participating in an evaluation of the Learner 
Response System (LRS). The aim of the project is to find out if using a hand held electronic 
learning device in the classroom is an effective way of improving maths and literacy. 


What is a Learner Response System and why is it being used? 


The LRS Project is being organised by Edge Hill University. It involves the use of pupil 
hand-held electronic devices known as ActivExpression. Each pupil in a classroom is given a 
handset that has a series of buttons and a small screen. Pupils can answer questions and 
receive instant feedback via the handset. When pupils respond to a question using the 
device, teachers can see on their own linked computer screen how individual pupils 
answered: for example, how long they are taking with each question, how many attempts 
they need to get it right, where they are going wrong. This provides information to the 
teacher on which elements of a topic pupils understand or find challenging, so that they can 
adjust their approach within the lesson and target support at struggling pupils. It is also 
intended to be fun and engaging. 


How will we know if the LRS approach improves learning? 


We do not know whether using the hand-held electronic device helps learning. To find out, a 
research team from the UCL Institute of Education are evaluating the impact of the LRS 
approach. Around 9000 children from 100 schools are taking part in the project. Fifty schools 
were chosen at random by the research team to receive the LRS approach. The other 50 
schools (of which your child’s school is one) are a comparison group and have their usual 
lessons. 


To find out whether the LRS programme makes a difference, the evaluation team would like 
to follow the current Year 5 and Year 6 students’ achievement in English and Maths SATS, 
and compare these between the schools that have the programme and those in the control 
schools. To do this, we will need to have the students’ maths and English SATs scores at 
Key Stage 1 and Key Stage 2. We will ask schools to provide their students’ names and 
unique pupil numbers so that we can access these tests scores from the National Pupil 
Database. All pupil data will be treated with the strictest confidence. We will not use your 
child’s name or the name of the school in any report arising from the research. 


What will happen to the information after the study finishes? 
The data from the study will be anonymised (all names removed) and shared for research 
purposes with two special data archives being kept from all the education projects being 
carried out for the Education Endowment Foundation — the EEF Data Archive and the UK 
Data Archive 

What happens next? 
If you or your child would like more information about what is involved in taking part in the 
LRS research project, please feel free to contact Meg Wiggins, Director of the Evaluation, 
whose details are below. 


| “UCL — Fail A Education 


University Endowment 
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LEARNER RESPONSE SYSTEM (LRS) Project Evaluation 


If you are happy for your child to take part in the study you do not need to do anything. If 
you do not want your child’s results to be included, please complete the attached ‘opt 
out’ form and return it by post to the research team at the address given below by the 
end of June 2015. Opting out of the evaluation will not affect your child’s time in schoolin 
any way or the lessons they receive. You are free to ask for your child to be withdrawn from 
the evaluation at any time. 


You will be able to find out the results from the project once it has been completed in 2017 
from the following websites: www.edgehill.ac.uk; www.ioe.ac.uk; and 


www.educationendowmentfoundation.org.uk. 


We look forward to working with your son/daughter’s school and value your child’s 
contribution to the Learner Response System evaluation. 


Yours faithfully, 


Meg Wiggins (IOE Evaluation Director) 
m.wiggins@ioe.ac.uk or 0207 612 6786. 


Janice Jackson (Edge Hill): Jacksoni@edgehill.ac.uk 


Opt- out form — ONLY COMPLETE IF YOU DO NOT WANT YOUR CHILD’S TO BE INCLUDED IN THE 
LEARNER RESPONSE SYSTEM PROJECT EVALUATION. 


Child’s name: DOB: 
School name: 
Please tick all that apply: 


| do not want my child’s anonymous exam results to be included in the Learner Response System 
Project evaluation. 0 


Signed: 


Print name: 
Relationship to child: 


Return to: Meg Wiggins, Social Science Research Unit, UCL, Institute of Education 
18 Woburn Square, London WC1H ONR 


ET Edge Hin A Eee 


University Endowment 
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Appendix C: Case Study Site — Children’s focus group 
schedule 


School ID 
Date/time 


Note down: Year group of participants/ gender of participants/ what have children 
missed to attend this group? 


Researcher name/any other adults present 
Introduce myself — name, researcher from big university in London. 
First of all - what do you call the PODS? 


I’m Visiting 6 schools in different parts of England and want to find out what 
children think of the [PODS]. So, | am talking to about 30 children like you. 


Schools have to buy lots of things e.g. computers, reading books, sports equipment 
etc. My job is to help schools and the government decide if the [PODS] are a good 


thing to spend money on in the future. 


Are you still happy to be part of the group? You don’t have to take part if you 
don’t want to. 


Are you happy for me to tape our discussion? (Maybe let them say their teachers 
name and then play back so that they can hear — as icebreaker?). No one will listen 
to the tape except me and my university team. No one in the school will find out what 
you have said about the [PODS] today. If not happy - fine 


Have you got any questions before we start? 


| won't mention your name when | talk about what children have told me about the 


PODS...but it would help me while we are talking today to know your names so can 
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you tell me all your name one by one and | will write it down but | will throw my piece 


of paper a way at the end of the group 


So.... 


1. | want to know is whether you like using the [PODS] or not 


Please can you write down on this piece of paper whether you think the 
[pods] are REALLY GOOD, OK, NOT GOOD? 


Now can you hand me your pieces of paper. 


If all said REALLY GOOD ask — 
Is there anything you DON’T like about them?. 


If no response ask 


1. Have you always liked them or at the beginning you were less sure? 


2. Is there anyone in your class who doesn’t seem to like using them as much as you? 


If said OK or NOT GOOD ask — 
What stopped you answering ‘really good’? — 


What is it that you don't like? 
Do you prefer lessons when the [PODS] are not used? 


2. What lessons do you like using them best in? worst in... 


3. Ask for examples of exercises they have done — 


probe for working as a group etc. 


4. What is different about a lesson where you use the [pods] and one where you 


don’t? 


5. Imagine you never had a lesson again with the [pods] how would you feel? 


6. Now I would like to know if you think they help children to learn? Please can 
you write down on this piece of paper — YES, No or NOT SURE 
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Depending on answers ask.... 


Why do you think they help? — what is different about the way the teacher teachs 
you with the [PODS]? 


...prompt for quality and speed of feedback; prompt for impact on maths and literacy 


When do you think they DON’T help? — Can you think of a time when you don’t 
think they helped you or someone else in the class? 


THANK YOU VERY MUCH! 


After group — note.... 


Range of learning ability in the group 
Ethnicity 


Views on participation by different members of the group etc... 
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Appendix D: Case study site interview schedule 


(note: adapt for role of interviewee) 


Date of interview 


School ID 

Staff member role 

Verbal consent to take part 
Consent to record 


Confidentiality 


*How many and what ‘type’ of staff are using the LRS SMT/ 
(pods) in this school? Has its use been extended beyond Teacher/TA 
the original trained staff; if yes how have new staff been 

trained? 

“How much is LRS being used by you — ie in what Teacher/TA 


proportion of your lessons (total and maths/literacy 
lessons) do you use the 

pods and for what proportion of the lesson. 

At what level are you using it? basic .....advanced? How Teacher/TA 
does this compare with your colleagues? If basic do you 
think you will progress to advanced? — 


“What do you like about using it? Teacher/TA 
“What do you not like about using it? SMT (change 
Do you have a preference for using it in different subjects? | to make 

Has your view changed over the year? How does your view__| appropriate) 
compare with your user colleagues? 


“How do you use it most (e.g. ad hoc questions, core 


resources, your own resources etc) Do you think your 
colleagues are the same as you — or do you think they use 


it differently? 


“What kind of feedback do you use? E.g. do you show the 


results to the whole class, do you keep results anonymous 
and feedback 


1:1 — why do you use the approach you use? 


“Have there been any particular challenges? E.g. IT 
issues, time for training/learning/preparing resources, 


reaction of colleagues 
— what has facilitated you embedding the use of the pods in 
your teaching? 
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*Do you think it will have an impact (over and above using 
more traditional teaching approaches) on KS2 results? 


Have you seen any impact on results so far? If yes to impact 
— why do you think this is? Eg improved/faster feedback 


Teacher/TA:SM 
T 


*If no impact, why? Barriers to impact e.g. feedback is too blunt a 
tool, relies on enthusiasm/skill of the teacher/IT ability of teacher 
etc 


Teacher/TA 


*Do the children enjoy using LRS — is this all children/some more 
than others? Gender, ability, language 


Teacher/TA 


What are your views on Edge Hill’s support? Include training 
sessions; on site support. How will it be in future without this 
support? Any issues with monitoring activities etc? 


*Will LRS use be continued in the school next year? If so with 
which year groups and which teachers? What ongoing 
support/training do you think will be required for current users 
and new users? 

If yes — what are the costs? How will these be met? Would they 
choose implementing LRS over other interventions? What 
barriers to on-going use do you envisage? 


If no — not continued — why? 


SMT/Teacher 


How does LRS experience compare with other new teaching 
interventions they have come across? 


SMT/Teacher 


Would they recommend it to other schools? Should it become a 
part of primary school education across the UK? 


SMT/Teacher 


Thank you 
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Appendix E: LRS Intervention Teachers — year 1 survey 


GETTING STARTED 


Thank you for taking the time to complete this questionnaire about the Edge Hill Learner Response System 


(LRS) programme this year. 


As you are aware the LRS hand held devices are called many different things by different schools; in this 


questionnaire we have called them 'pods'. 


This questionnaire is intended mainly for class teachers who are responsible for a year 5 or 6 class. If you 
are nota class teacher but have been one of the main people in your school using the LRS with year 5/6 pupils 


then we would be grateful if you would complete the questionnaire. 


All the information you give in the questionnaire will be confidential - only the UCL Institute of Education 
research team will see your responses - they will not be shared with your school or with Edge Hill University. 
Any information published from answers that you give will be anonymised - your name will never be used. By 


filling in the survey, we will assume that you are providing consent for us to use your answers in the evaluation. 


* 1. To get started, please enter your three digit survey code from the email here. 


survey code 


LRS teachers intervention 


ABOUT YOU 


2. Since qualifying, how many years have you been teaching (either full or part time)? 


io 1-2 years 
OC) 3-5 years 


) 6 years or more 


3. What is your gender? 


C) Female 


C) Male 


4. Which age group are you in? 
C) 20-30 years 
C) 31-40 years 
OC) 41-50 years 


C) 51 years or over 


5. Which of the following are you? (Please tick all that apply) 


A year 5 class teacher 


A year 6 class teacher 


|__| Acombined year 5/6 class teacher 


A head teacher or deputy head 


‘— Other (please explain in the comment box) 


Comments 


6. How would you rate your level of IT skill at the start of this school year? 


+) Very high 


*) Fairly high 


() Fairlylow 


) Very low 
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ABOUT YOUR CLASS 


In this section we ask for information about your class during the past school year. We realise that there is often 
mobility, with children joining and leaving the school throughout the year - so please give average figures where 


werequest numbers. 


If you do not have your own class (e.g. you are a deputy head who also teaches year 5/6 at times) please 
answer the questions in this section for the class you have mainly used the pods with this year. There is a 


comments box in question 12 at the end of the section for you to add any explanationifrequired. 


7. How many pupils have you had in your class (on average) during the past year? 


OC) 25 orless 
() 26-30 


C 31 and over 


8. How many girls and how many boys are in the class (on average)? 


Girls 
Boys 


9. How many pupils on average in your class have additional needs that affect their learning (for 


example English as asecond language, special educational needs, etc)? 


10. How many pupils in your class are categorised as gifted and talented? 
Number of gifted and talented pupils 


11. How many adults (other than you) are routinely in your classroom for at least some of an average week 


(TAs, mentors for children with additional needs etc) ? 


12. Is there anything extraordinary about your class that you'd like to share with us? 
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TRAINING AND SUPPORT - LRS PROGRAMME 


This section includes questions about the training and support offered for the Learner Response System. 


13. If you missed any of the training sessions why was this? (please tick all that apply) 


I started late on the programme 


| was sick or had family emergency 


| had other school commitments that | decided were a higher priority 


There were other school commitments that the SMT decided were a higher priority 


| wasn't made aware of the training dates 


| | forgot about the training 


| didn't find the training useful 


'—— Other (please explain in comments box) 


Comments 


14. How many of the Edge Hill training days have you attended? 


How many of the two baseline training days in June/July 2014 did you attend? (please type in: 0, 1 or 2) 


Po 


How many of the five half termly training days did you attend? (please type in: O - 5) 


15. In addition to the training days, how much other support and training did you receive from the Edge 


Hill team? 


How many visits did members of the Edge Hill team (e.g. Jan, Pat) make to your school? (exclude any training 


days) 


eee 


Approximately how many times over this school year did you contact the Edge Hill team by email/phone for 


advice or as 


16. How would you rate the training and support you have had from the Edge Hill team? Please tick one on each 


line. 
Excellent Good Quite good Poor Notapplicable 
2 day baseline training C) C) C) C) C) 
5 half termly training C) (>) i) (>) () 
VA es V4 V4 V4 
days 
School visits by members ~~ o~ o~ a a 
( ) ( @ ) ( 
of the Edge Hill team bal /} \/ VJ di 
Email or telephone 
support CD) iw) C) t) Ks) 


Please comment on the training and support role of Edge Hill if you would like to 
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YOUR USE OF THE LRS PODS IN MATHS LESSONS 


The next questions are about your use of the LRS pods SINCE JANUARY in maths lessons. We appreciate that 
usage will have varied from week to week (for example before and after SATs) - so please imagine an average 


week for your class when deciding on your answer. 


17. On average, how often since January, have you used the LRS pods to teach maths? 
C) More than once aschool day 


ca Once a school day 
OC) Several times a week 
C) Once a week 

OC) Once afortnight 
C) Once or twicea term 


Never 


O 
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18. Inan average week, approximately how many minutes have you used the pods for in maths lessons with 


yourclass? 
C) 1 - 30 minutes 


OC) 31 - 59minutes 


OC) 1-3 hours 


C) More than 3 hours but less than 6 hours 


CD) More than 6hours 


19. Inwhichterm this year have you used the pods most frequently for teaching maths? 


OC) Autumn 
C) Spring 
C) Summer 


C) All equally 


20. Which of the following are the main ways you use the pods in teaching maths? (please tick all that apply) 


At the beginning of the day as children are arriving in the classroom 


|__| Asawarm up for the whole class at the beginning of a lesson 


As the main focus of lessons, for the whole class 


On and off throughout the lesson, with the whole class 


As a tool for working with small groups of children 


As a tool for individuals working in pairs with peers or with a member of staff 


As a tool for individuals working on their own 


|__| Other (please explain) 


Please add any comments on these and any other ways you use the pods for maths 


21. Which of the following best describes your current ability level for using the pods in maths teaching? 
Basic level use (i.e. you are only able to use the system in a limited range of ways) 
Intermediate level use (i.e. you are able to use most of what you have been taught in the Edge Hill training) 


Advanced level use (i.e. you are able to use most of what you have been taught in the Edge Hill training plus additional self- taught 
usage) 


22. In general which of the following do you use in your maths teaching? (Tick all that apply). 
The core resources - unadapted 
The core resources - adapted by you 
Resources that you have made 
Resources that other teachers have made 


Learning Clip 


INSIDE 


Other 


please explain other usage for maths 


23. In your experience do children across the academic ability spectrum enjoy using the pods in maths? (Please 


tick one answer on each line) 


Yes, nearly all really like © Some enjoy using them No, many don't enjoy 


using them but by no means all using them 


Fastest learners 


Moderate learners 


Slower learners 


Children with SEN 


If you have answered that children don't enjoy using the pods please tell us more about this here 


24. Do you think using the pods has had a positive impact on maths attainment across the ability 


spectrum? (Please tick one on each line) 
Yes Maybe No 
Fastest learners 


Moderatelearners 


Slower learners 


Children with SEN 


25. If there is anything that we have not covered about your use of the pods in maths teaching in this 


questionnaire then please use this box to tell us about this. 


26. If you have never or rarely used the pods in maths, please explain why. (If you have used it more 


frequently, skip to the next question) 


27. In addition to the Learner Response System, this year has your class had any special additional support to 
boost maths attainment? (e.g. an external programme, extralT or other equipment, extra staff resources, extra 


class time etc) 


( ~) Yes 


If yes, please explain 
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YOUR USE OF THE LRS PODS IN TEACHING LITERACY 


The next few questions are about your use of the pods in teaching literacy. As with the questions on maths 


lessons please imagine an average day/week for a year 5 or year 6 class when choosing your answer. 


28. On average, how often since January, have you used the pods to teach literacy? 
OC) More than oncea day 

OC) Once a day 

OC) Several times a week 

C) Once a week 

OC) Once afortnight 


C) Once or twicea term 


OC) Never 
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29. How many minutes, approximately, inan average week, have you used the pods for literacy lessons with 


yourclass? 
C) 1 - 30 minutes 
C) 31 -59 minutes 


C) 1-3 hours 


More than 3 hours but less than 6 hours 


I'm not using the pods in literacy 


V) 
C3 More than 6hours 
wy, 


30. In which term this year have you used the pods most frequently for teaching literacy? 


C) Autumn 
C) Spring 
OC) Summer 


ie All equally 


31. Which of the following are the main ways you use the pods in teaching literacy? (please tick all that apply) 


At the beginning of the day as children are arriving in the classroom 


As a warm up for the whole class at the beginning of a lesson 


|__| As the main focus of lessons, for the whole class 


On and off throughout the lesson, with the whole class 


‘| As a tool for working with small groups of children 


As a tool for individuals working in pairs with peers or with a member of staff 


| Asa tool for individuals working on their own. 


Please add any comments on these and any other ways you use 


32. Which of the following best describes your current ability level for using the pods in literacy 


teaching? 
Basic level use (i.e. you only use the system ina limited range of ways) 
Intermediate level use (i.e. you are able to use most of what you have been taught in the Edge Hill training) 


Advanced level use (i.e. you are able to use most of what you have been taught in the Edge Hill training plus additional self- taught 
usage) 


33. In general which of the following do you use in your literacy teaching? (Tick all that apply). 


The core resources - unadapted 


The core resources - adapted by you 
Resources that you have made 


Resources that other teachers have made 
Learning Clip 


INSIDE 


Other please explain 


Please add a comment if you wish 


34. In your experience do children across the academic ability spectrum enjoy using the pods in 


literacy? (Please tick one answer on each line) 


Yes, nearly all really like Some enjoy using them No, many don't enjoy 


using them but by no means all using them 


Fastest learners 


Moderatelearners 


Slower learners 


Children with SEN 


35. Do you think using the pods has had a positive impact on literacy attainment with your pupils? Please 


answer for whether this is the case across the ability spectrum. (Please tick one on each line) 


Yes Maybe No 


Fastest learners 


Moderatelearners 


Slower learners 


Children with SEN 


36. If there is anything that we have not covered about your use of the pods in literacy teaching in this 


questionnaire then please use this box to tell us about this. 


37. If you have never or rarely used the pods in literacy teaching, please tell us why. (If you have used more 


frequently, youcan skip to the next question) 


38. In addition to the Learner Response System, this year has your class had any special additional support to 
boost literacy attainment (e.g. an external programme, extralT or other equipment, extra staff resources, extra 


class time etc) 


( ~) Yes 


If yes, please explain 
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FINAL QUESTIONS 


We have a few final questions about your experiences and views. 


39. Have you used the pods to teach pupils subjects other than maths and literacy? 


C) Yes, often 


Cal Yes, sometimes 


C) No, never 


40. Compared with other teaching methods you have used, how do you rate the LRS in literacy, maths and 


other subjects? (please tick one on each line) 


Ok but I don't think 
it adds anything to 


Excellent, | see it Generally good and my teaching that 
as an essential avaluable addition othermethods can't 
teaching asset. to my teaching provide 


Maths ©) O) ©) 


Literacy C) C) 
Other subjects ©) C) C) 


Poor, it has not 
worked well for 


me/my class 


O) 


Notapplicable 


e 
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41. Have other staff in your school (in addition to you and your year 5 and 6 colleagues) received training inthe 
LRS? 


C) Yes 
C) No 
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42. Who provided thistraining? 


Edge Hill staff 


School staff 


Other 


Other (please specify) 


43. Which types of staff received this training? 


Headteacher 


Other senior school staff 


Other class teachers HLTAs 


r— TAs 


Other 


Other (please specify) 


44. Which of the following do you use (generally and when using the pods) to give individual pupil's 
feedback on their work? Please tick all that apply. 


Privately on a 1:1 level with a pupil 


To the whole class but anonymously (so individuals can identify their own work but others aren't aware whose answers are whose) 


To the whole class not anonymously 


Other (please explain in the comments box) 


We are interested in your comments on why you use the approaches you have ticked 


45. Who decides the approach to pupil feedback in your class? Tick all that apply 


There is a school policy on feedback 

You, the class teacher decides what kind of feedback to give 
The children are asked individually how they would like feedback 
The children are asked as a class how they would like feedback 


Please comment if you wish 


46. How do you use the results from the LRS to evidence progress and curriculum coverage? Tick all that apply 
Stick printouts from the system in pupil's books 
Export to an electronic central file 


Print out paper copy and store in a class assessment file 


Print out paper copy and put on a notice board in the classroom 


Other 


Other(please-specify) 


47. Towhat extent, if at all, have the following proved challenging in terms of the LRS programme? (please tick 


all that apply) 


Somewhat Not at all 


Very challenging challenging challenging 
Finding the time to prepare resources 


Finding the time to learn the new IT skills required 


Not having sufficient IT skills and confidence 


Practical problems with the technology and equipment 


Amount of support from the head/SMT at my school 


Amount of support from the lead person at my school for the LRS 


programme 


Amount of support from other school KS2 colleagues also 


assigned to the LRS programme 


Amount of support from school colleagues not assigned to the programme 


Adapting to make it appropriate for effective use with your class 


Staff turnover 


Meeting the minimum requirements set by Edge Hill 


Please tell us about any other challenges not listed here 


48. The following statements have been made by some teachers who have used the LRS and the developers 


of the system. From your own experience of using it, how much do you agree or disagree with these 


statements? (Please tick one on each row) 


It makes practicing through repetition more feasible for pupils 
Showing LRS results to the whole class is motivating for pupils 


Teachers can assess and intervene more quickly when pupils are 


struggling 
Teachers can plan lessons more effectively 


It facilitates sharing of work/ideas between pupils 
It facilitates engagement by all pupils 


It promotes independent learning through instant feedback on the 


pod 
It allows pupils to work at their own pace 


It represents an approach to education that is potentially 


undesirable 


Strongly 


Strongly agree Agree Disagree disagree 


Please add any other aspects of the LRS- positive or negative- that you think are important 


49. How does the school intend to use the pods next year? (please tick one on each line) 


Definitely Possibly No Don't know 


Continue with use in Year - ~ = z 
6 ONLY 


Continue with use in Year 
Sand 6 


Extend use to other year 


groups 
Discontinue the programme 


Other (please specify) 


50. Which of the following most closely matches your own plans for use of the pods in the future? 


lam satisfied using the pods at my current level and don't expect to advance my level of use significantly 
| would like to become more advanced in my use of the pods and expect to become so in the future 
| would like to become more advanced in my use but expect in reality | will continue using them at my current level 


| don't expect to use the pods in the future (at this school) 
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51. Why do you not expect to use the pods at this school next year (please tick all that apply)? 


lam leaving this school 


The school is not continuing with the programme 


| am teaching a year group that will not be using the pods 


| have not enjoyed using the pods and have chosen not to continue with them even though the programme is continuing in the school 


Other (please explain below) Please 


add explanations and comments 


52. If you would like to tell us anything else about the LRS programme please do so here 
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THE END 


Many thanks for completing this questionnaire! 


Appendix F: LRS Intervention Teachers — year 2 surveyLRS 


GETTING STARTED 


Thank you for taking the time to complete this questionnaire about the Edge Hill Learner Response System (LRS) 
programme this year. AS you are aware the LRS hand held devices are called many different things by schools; in 
this questionnaire we have called them 'pods'. All the information you give in the questionnaire will be confidential 
- only the UCL Institute of Education research team will see your responses - they will not be shared with your 
school or with Edge Hill University. Any information published from answers that you give will be anonymised - 
your name will never be used. 


* 1. To get started, please enter your three digit survey code from the email here. 


ABOUT YOU 


2. Since qualifying, how many years have you been teaching (either full or part time)? 


() 1-2 years 
(.) 3-5 years 


\ 6 years or more 


3. What is your gender? 
(_) Female 


©) Male 


4. Which age group are you in? 
O 20-30 years 
(_) 31-40 years 
O 41-50 years 


(.) 51 years or over 


5. Which of the following are you? (Please tick all that apply) 
A year 6 class teacher 


| | A combined year 5/6 class teacher 


A head teacher or deputy head 


'_ | Other (please explain in the comment box) 
Comments 


6. How many school years have you personally been part of the LRS programme? 
O 2 years 


O 1 year 
O other 


TRAINING AND SUPPORT - LRS PROGRAMME 


This section includes questions about the training and support offered for the Learner 
Response System. 


7. How useful did you find the different aspects of the LRS training? 


Very useful Quite useful Not very useful Not at all useful 
Sharing LRS 


experiences with C) C) OC) O) 


other schools 


Dedicated time away ~ a vn en 
from school to J UU VU) WU) 
practice LRS skills 


Advice and support C) C) C) C) 
from Edge Hill team 


Learning new CU) O O UO 
aspects of the LRS 


8. If you missed any of the training sessions this academic year, why was this? 
(please tick all that apply) 


[] | started late on the programme 


| was sick or had family emergency 


/ | | had other school commitments that | decided were a higher priority 


There were other school commitments that the SMT decided were 
___ ahigher priority 


__| | wasn't made aware of the training dates 


|__| | forgot about the training 


__| I didn't find the training useful 


| didn't feel | needed the training 


Other (please explain in comments box) 


Comments 


9. What is your view on the amount of training you had from the EdgeHhill team 
this academic year? Was it... 


© About right 


© Too much 


©) Too little 


Comments 


YOUR USE OF THE LRS PODS IN MATHS LESSONS 


The next questions are about your use of the LRS pods this academic year in maths lessons. We appreciate that 
usage will have varied from week to week (for example before and after SATs) - so please imagine an average 
week for your class when deciding on your answer. 


10. On average how often this year have you used the LRS pods in your maths 
teaching? 


\” More than once a school day 
\ Once a school day 

\/ Several times a week 

{ 


\/ Once a week 


© Once a fortnight 


O Once or twice a term (please explain why you have used the pods so rarely 
in the comments box) 


VU Never (please explain why you have never used the pods in the comments 
box) 


11. In an average week, approximately how many minutes have you used the pods 
for to support maths learning with your class? 


() 1-30 minutes 

(©) 31 - 59 minutes 

© 1-3 hours 

\ More than 3 hours but less than 6 hours 
—) More than 6 hours 


©) lam not using /only rarely using the pods in maths lessons 


12. Which of the following are the main ways you have used the pods in teaching 
maths? (please tick all that apply) 


At the beginning of the day as children are arriving in the classroom 


As a warm up for the whole class at the beginning of a lesson 
As the main focus of lessons, for the whole class 


On and off throughout the lesson, with the whole class 


At the end of a lesson to assess understanding 


As a tool for working with small groups of children 


As a tool for individuals working in pairs with peers or with a member of staff 


Other (please explain) 
Please add any comments on these and any other ways you use the pods for maths 


13. Which of the following best describes your current ability level for using the 
pods in maths teaching? 


(_) Basic level use (i.e. you are only able to use the system in a limited range of ways) 


(_) Intermediate level use (i.e. you are able to use most of what you have been taught in the 
Edge Hill training) 


(_) Advanced level use (i.e. you are able to use most of what you have been taught in the 
Edge Hill training plus additional self- taught usage) 


14. How frequenily did you use the pods in your teaching of the following aspects of maths? 


Very often Quite often Occasionally Rarely/never 
Algebra C) () C) C) 
Geometry O O O O 
Graph and Data C) C) C) C) 
Measure O O O O 
Mental Maths C) C) C) C) 
Number ‘2 O O @) 
Statistics C) C) C) C) 


15. For which two components of the maths curriculum have you found the pods most useful in 
your teaching? (Please write in - e.g. fractions) 


1 fT 
2. 


16. In addition to the Learner Response System, this year has your class had any special additional 
input to improve maths attainment? 


No - no special additional input 


__| Yes - online maths learning software 


__ Yes - booster groups within lesson time 


|_| Yes - booster groups outside of lesson time 


Yes - additional teacher/TA support with maths in the classroom 


|_| Yes - a maths specialist visiting the school 


Yes - other, please explain in box 


If yes, please explain 


17. If there is anything that we have not covered about your use of the pods in 
maths teaching in this questionnaire, then please use this box to tell us about this, 


YOUR USE OF THE LRS PODS IN TEACHING ENGLISH 


The next few questions are about your use of the pods to support your English teaching. As with 
the questions on maths lessons please imagine an average day/week for a year 6 class when 
choosing your answer. 


18. On average how often this year have you used the LRS pods in your English teaching? 


~) More than once a school day 
~) Once a school day 

-) Several times a week 

+) Once a week 


>) Once a fortnight 


>) Once or twice a term (please explain why you have used the pods so 
~ rarely in the comments box) 
) Never (please explain why you have never used the pods in the comments box) 


Please comment here 


19. How many minutes, approximately, in an average week, have you used the pods for supporting 
English teaching with your class? 
_-) 1-30 minutes 
) 31 -59 minutes 
1-3 hours 
_-) More than 3 hours but less than 6 hours 
_-) More than 6 hours 


©) I'm not using the pods/using the pods very rarely in English lessons 


20. Which of the following are the main ways you use the pods in teaching 
literacy? (please tick all that apply) 


At the beginning of the day as children are arriving in the classroom 


As a warm up for the whole class at the beginning of a lesson 


As the main focus of lessons, for the whole class 


On and off throughout the lesson, with the whole class 
At the end of a lesson to assess understanding 


As a tool for working with small groups of children 


As a tool for individuals working in pairs with peers or with a member of staff 


Other (please explain) 


Please add any comments on these and any other ways you use the pods for English 


21. Which of the following best describes your current ability level for using the 
pods in English teaching? 


(_) Basic level use (i.e. you only use the system in a limited range of ways) 


(_) Intermediate level use (i.e. you are able to use most of what you have been taught in the Edge 
Hill training) 


(_) Advanced level use (i.e. you are able to use most of what you have been taught in the 
Edge Hill training plus additional self- taught usage) 


22. How frequenily did you use the pods in your teaching of the following aspects of English? 


Very often Quite often Occasionally Rarely/never 

Spelling C) C) C) C) 
Punctuation O O O O 
Word classes C) C) C) C) 
Grammar O O) O O 
Sentence level OQ O O OC 
Reading = _ _ - 

comprehension i} O O O 
Writing e O O O 


23. For which two components of the English curriculum have you found the pods 
most useful in your teaching? (Please write in - e.g. punctuation) 


1. CS 
2 


24. In addition to the Learner Response System, this year has your class had any 
special additional input to improve English attainment? 


No - no special additional input 


|_| Yes - online English learning software 


Yes - booster groups within lesson time 


|_| Yes - booster groups outside of lesson time 


Yes - additional teacher/TA support with English in the classroom 


|| Yes-a literacy specialist visiting the school 


Yes - other, please explain in box 


If yes, please explain 


25. If there is anything that we have not covered about your use of the pods in English teaching in 
this questionnaire then please use this box to tell us about this. 


THE FUTURE OF THE PODS IN YOUR SCHOOL 


We have a few questions about the future use of the pods in your school 
26. Does the school intend to use the pods next year? 

() Yes, definitely 

O Yes probably 

O Don't know 


() No (please tell us why this is the case in the comments box) 


Comments 


27. Which year groups is it likely to be used with next year? (Please tick all that apply) 


|_| Year 6 


Year 5 


__| Year 4 


Year 3 


__| Year 2 


Year 1 


L] Reception 


28. Do you think you/your LRS trained colleagues would be able to provide training in the use of 
the pods for teachers in the school if there is not on-going external training support? 


O Yes 
O No 
O Not sure 


O Not applicable 


FINALLY 


29. Compared with other teaching tools you have used how do you rate the LRS in 


literacy, maths and other subjects? (please tick one on each line) 


Excellent, 
| see it as 
an 
essential 
teaching 
asset 


Maths C) 


Literacy CQ) 
Other subjects C) 


Generally 
good and 
a valuable 
addition to 
my 
teaching 


6 


OK but | don’t 
think it adds 
anything to my 
teaching that 
other methods 
can't provide 


Poor, it 
has not 
worked 
well 
me/my 
class 


Not 
applicable 


If you have answered that you do not rate the LRS highly please explain why here 


30. Whai, in your opinion, are the main benefits of the LRS? 


31. What, in your opinion, are the main drawbacks of the LRS system? 


32. If there is anything else you would like to add about the LRS programme please 


Thank you very much for completing this questionnaire! The results of the LRS 
evaluation will be available in 2017. 


Appendix G: EEF cost rating 


Cost ratings are based on the approximate cost per pupil per year of implementing the intervention over 
three years. More information about the EEF’s approach to cost evaluation can be found here. Cost 
ratings are awarded as follows: 


Very low: less than £80 per pupil per year. 
Low: up to about £200 per pupil per year. 


Moderate: up to about £700 per pupil per year. 


Very high: over £1,200 per pupil per year. 


High: up to £1,200 per pupil per year. 


Appendix H: Security classification of trial findings 


Maths Cohort A 


hoe : iti : Final 
Criteria tor rating 
Criteria for ratin Adjust 
Design| Power | attrition 
0-10% 


Fair and clear quasi- 
experimental design for 


comparison (e.g. RDD) Adjustme 
with appropriate analysis, | MDES < nt for 
or experimental design 0.3 Balance 
with minor concerns about [-1] 
validity 


3a | Well-matched comparison 
(using propensity score 
matching, or similar) or MDES < 7 
experimental design with 0.4 21-30% 
moderate concerns about 
validity Adjustme 
2@ |Weaklymatched § [| #| &4'[| J J ne ae : i 
comparison or MDES < ‘ 
experimental design with 0.5 31-40% eit | 
major flaws 
ee [0] peers 
18 Comparison group with 
poor or no matching (E.g. | MDES < 51-50% 
volunteer versus others) 0.6 
Oe 
No comparator a > | S590% 


Initial padlock score: lowest of the three ratings for design, power and attrition = 5 padlocks 
due to 11% attrition 

Reason for adjustment for balance (if made): there is an imbalance of 0.14 sd between the 
two groups at baseline (average KS1 point score). This is mainly due to poor matching at the 
NPD level due to poor quality of school level data and not likely due to differential attrition. 
Though it should be noted that there was differential attrition prior to collection of pupil level 
data post randomisation. 

Reason for adjustment for threats to validity (if made): none 

Final padlock score: initial score adjusted for balance and internal validity = 3 padlocks 


Maths Cohort B 


Criteria for rating 


Fair and clear quasi- 
experimental design for 
comparison (e.g. RDD) 


with appropriate analysis, | MDES < 11-20% 
or experimental design 0.3 
with minor concerns about 
validity 

38 | Well-matched comparison 
(using propensity score 
matching, or similar) or MDES < 5 
experimental design with 04 21-30% 
moderate concerns about 
validity 

28a Weakly matched 
comparison or MDES < 5 
experimental design with 05 31-40% 
major flaws 

18 Comparison group with 
poor or no matching (E.g. | MDES< | 51 500, 
volunteer versus others) 0.6 

0a 
No comparator rere > | S500% 


Adjustme 
nt for 
Balance 


[0] 


Adjustme 
nt for 
threats to 
internal 
validity 


[0] 


Initial padlock score: lowest of the three ratings for design, power and attrition = 5 padlocks, 


6.9% attrition 


Reason for adjustment for balance (if made): none 


Reason for adjustment for threats to validity (if made): none 
Final padlock score: initial score adjusted for balance and internal validity = 5 padlocks 


Reading Cohort A 


OTgi lat Mcolme-lilare| 


Attrition* 


5a 


Fair and clear quasi- 
experimental design for 
comparison (e.g. RDD) 
with appropriate analysis, 
or experimental design 
with minor concerns about 
validity 


MDES < 
0.3 


0-10% 


38 | Well-matched comparison 
(using propensity score 
matching, or similar) or MDES < 5 
experimental design with 04 21-30% 
moderate concerns about 
validity 

2a Weakly matched 
comparison or MDES < 5 
experimental design with 05 31-40% 
major flaws 

18 Comparison group with 
poor or no matching (E.g. | MDES< | 51 500, 
volunteer versus others) 0.6 

0a 
No comparator arr > | S500% 


Adjustme 
nt for 
Balance 


[-1] 


Adjustme 
nt for 
threats to 
internal 
validity 


[0] 


Initial padlock score: lowest of the three ratings for design, power and attrition = 5 padlocks 


due to 12% aittrition 


Reason for adjustment for balance (if made): there is an imbalance of 0.14 sd between the 
two groups at baseline (average KS1 point score). This is mainly due to poor matching at the 
NPD level due to poor quality of school level data and not likely due to differential attrition. 
Though it should be noted that there was differential attrition prior to collection of pupil level 


data post randomisation. 


Reason for adjustment for threats to validity (if made): none 
Final padlock score: initial score adjusted for balance and internal validity = 3 padlocks 


Reading Cohort B 


Org lat MColme-lilare| 


Fairandclearquasi | | | | fo fo 
experimental design for ; 
comparison (e.g. RDD) rel 
with appropriate analysis, | MDES < 11-20% nt for 
or experimental design 0.3 Balance 
with minor concerns about [0] 
validity 
3@ |Well-matchedcomparison| |  £#4| | Jf fo 
(using propensity score 
matching, or similar) or MDES < 
experimental design with 0.4 21-30% 
moderate concerns about 
validity Adjustme 
ieee htter 
- ea is MDES threats to 
< °. ] 
experimental design with 05 31-40% pleMnal 
; validity 
major flaws 
1a : Fe ee] (ce | | [0] — 
Comparison group with 
poor or no matching (E.g. | MDES< | 64 590, 
volunteer versus others) 0.6 
Oa OO 
No comparator rere > | S500% 


e Initial padlock score: lowest of the three ratings for design, power and attrition = 5 padlocks, 
10.4% attrition 

e Reason for adjustment for balance (if made): none 

e Reason for adjustment for threats to validity (if made): none 

e Final padlock score: initial score adjusted for balance and internal validity = 5 padlocks 
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